Attachment 1928758 Details for Bug 2149663 – RBD test logs

RBD test logs

perf_multi_snap_rbd_log.txt (text/plain), 2.59 MB, created by Ishwarya Munesh on 2022-11-30 14:27:39 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Ishwarya Munesh

Created: 2022-11-30 14:27:39 UTC

Size: 2.59 MB

patch

obsolete

>Unable to find the authentication configuration at /root/ocpcluster/ocs-ci/data/auth.yaml, please refer to the getting started guide (https://ocs-ci.readthedocs.io/en/latest/docs/getting_started.html#authentication-config)
>Failed to fetch auth.yaml from ocs-ci-data
>Unable to find the authentication configuration at /root/ocpcluster/ocs-ci/data/auth.yaml, please refer to the getting started guide (https://ocs-ci.readthedocs.io/en/latest/docs/getting_started.html#authentication-config)
>Command stderr: error: the server doesn't have a resource type "OperatorSource"
>
>Failed to get resource: ocs-operatorsource of kind: OperatorSource, selector: None, Error: Error during execution of command: oc -n openshift-marketplace get OperatorSource ocs-operatorsource -n openshift-marketplace -o yaml.
>Error is error: the server doesn't have a resource type "OperatorSource"
>
>Number of attempts to get resource reached!
>Failed to gather version info!
>Traceback (most recent call last):
>  File "/root/ocpcluster/ocs-ci/ocs_ci/framework/pytest_customization/ocscilib.py", line 434, in gather_version_info_for_report
>    mods = get_version_info(namespace=ocsci_config.ENV_DATA["cluster_namespace"])
>  File "/root/ocpcluster/ocs-ci/ocs_ci/ocs/resources/ocs.py", line 200, in get_version_info
>    csv_name = package_manifest.get_current_csv(channel)
>  File "/root/ocpcluster/ocs-ci/ocs_ci/ocs/resources/packagemanifest.py", line 187, in get_current_csv
>    raise ChannelNotFound(
>ocs_ci.ocs.exceptions.ChannelNotFound: Channel: stable-4.12 not found in available channels: ['stable-4.11', 'stable-4.10']
>Failed to gather version details! The report of version mightnot be complete!
>============================= test session starts ==============================
>platform linux -- Python 3.8.13, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
>rootdir: /root/ocpcluster/ocs-ci, configfile: pytest.ini
>plugins: flaky-3.7.0, repeat-0.9.1, ordering-0.6, metadata-1.11.0, logger-0.5.1, marker-bugzilla-0.9.4, html-3.1.1
>
>----------------------------- live log collection ------------------------------
>01:08:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get clusterversion -n openshift-storage -o yaml
>collected 1 item
>
>tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py::TestPvcMultiSnapshotPerformance::test_pvc_multiple_snapshot_performance[CephBlockPool-512] 
>-------------------------------- live log setup --------------------------------
>01:08:37 - MainThread - ocs_ci.utility.utils - INFO  - testrun_name: OCS4-12-Downstream-OCP4-12-BAREMETAL-IPI-1AZ-RHCOS-3M-3W
>01:08:37 - MainThread - ocs_ci.utility.utils - INFO  - Retrieving the authentication config dictionary
>01:08:37 - MainThread - ocs_ci.utility.utils - WARNING  - Unable to find the authentication configuration at /root/ocpcluster/ocs-ci/data/auth.yaml, please refer to the getting started guide (https://ocs-ci.readthedocs.io/en/latest/docs/getting_started.html#authentication-config)
>01:08:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pods -o name
>01:08:37 - MainThread - ocs_ci.ocs.utils - INFO  - pod name match found appending rook-ceph-tools-749dfcffb9-jr87p
>01:08:37 - MainThread - ocs_ci.ocs.utils - INFO  - Ceph toolbox already exists, skipping
>01:08:37 - MainThread - tests2.conftest - INFO  - All logs located at /tmp/ocs-ci-logs-1669712909
>01:08:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: /root/ocpcluster/ocs-ci/bin/oc version --client -o json
>01:08:37 - MainThread - ocs_ci.utility.utils - INFO  - Existing client version (4.12.0-0.nightly-2022-11-23-055211) does not match configured version (4.12.0-0.nightly-2022-11-28-145121).
>01:08:37 - MainThread - ocs_ci.utility.utils - INFO  - Downloading openshift client (4.12.0-0.nightly-2022-11-28-145121).
>01:08:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: tar xzvf openshift-client.tar.gz oc kubectl
>01:08:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: /root/ocpcluster/ocs-ci/bin/oc version --client
>01:08:41 - MainThread - ocs_ci.utility.utils - INFO  - Deleted backup binaries.
>01:08:41 - MainThread - ocs_ci.utility.utils - INFO  - OpenShift Client version: Client Version: 4.12.0-0.nightly-2022-11-28-145121
>Kustomize Version: v4.5.7
>
>01:08:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get csv  -n openshift-storage -o yaml
>01:08:41 - MainThread - ocs_ci.ocs.version - INFO  - collecting ocp version
>01:08:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get clusterversion version -o yaml
>01:08:42 - MainThread - ocs_ci.ocs.version - INFO  - collecting ocs version
>01:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get namespace  -o yaml
>01:08:42 - MainThread - ocs_ci.ocs.version - INFO  - found storage namespaces ['openshift-cluster-storage-operator', 'openshift-kube-storage-version-migrator', 'openshift-kube-storage-version-migrator-operator', 'openshift-local-storage', 'openshift-storage']
>01:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-cluster-storage-operator get pod  -n openshift-cluster-storage-operator -o yaml
>01:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-kube-storage-version-migrator get pod  -n openshift-kube-storage-version-migrator -o yaml
>01:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-kube-storage-version-migrator-operator get pod  -n openshift-kube-storage-version-migrator-operator -o yaml
>01:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-local-storage get pod  -n openshift-local-storage -o yaml
>01:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pod  -n openshift-storage -o yaml
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - ClusterVersion .spec.channel: stable-4.11
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - ClusterVersion .status.desired.version: 4.11.2
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - ClusterVersion .status.desired.image: quay.io/openshift-release-dev/ocp-release@sha256:98c294a87ab2e794e313e2d7da8a63efcc01e3e850ad92281ae39b7e525696e6
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-cluster-storage-operator
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60c88dda1e58b5a6e8109afef2a837fe184362ada32f369784bd5cf2a34183f2 {'905756d19dd33f606a0e727d401e5c118d1668bda1c315fd5ebb5f5dafeed035'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7caf6ed642540c96e82faec6ea468bda65cf30490a7f36d84a4c821524ec5d9a {'7b5dee5bec4d0257627332abe1a4d7a3a9303a8bc4348e3db1609b037d9f69cb'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe8d8fd53bacfd5ac2e5d70e0040be4cd65cd5550cb6cbc1ef0bddb01372d4e {'1722cd69c878d15a52553f96b3d54879b04cbd48e08cba784ac54fc63665aba3'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c34ae00e6572e47420ae12278b05b7190da2ba34a8e649b01d7e89ff9a5316 {'a433ba6f9d6c32ee7b764023aefedd2bb44e53a9153261d4412952cbebe0b163'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-kube-storage-version-migrator
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae9915adc15077ed2466d5e8100c337834db34b84c23c77c59bd031a38c66a8a {'c0ff9f540c35e36aca183d10524c6ffab62486a10162e587c5fa4153e9ab40fb'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-kube-storage-version-migrator-operator
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48ad9408d754564627549bc8035ae681201a18bdeeb9fa234049c44e6c14581e {'79c3762e8e63d7f34bea38ad71fdc4afe2ce0b3efae1a1ef285f73ccf4b043b1'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-local-storage
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:d4978704638dfe4762d16eff89315ce1071e7cdcbdda46695ade1b7c8c22e371 {'registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:0c785dbfd8d6ccb96c848c51c8a95d7d8fa1a0c360d74b289bc7d01da00f1e27'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:e3dad360d0351237a16593ca0862652809c41a2127c2f98b9e0a559568efbd10 {'registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:9d9c95c485abf0c842ec6f7b0c82e05176b3f8351fe840ae5ffa4d91f0100fb7'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-local-storage-operator@sha256:f468319691b7e26e14186b719786fc41f2613f44d856a6a40b7dfe95a38469c9 {'registry.redhat.io/openshift4/ose-local-storage-operator@sha256:78ae587f6de5f908a1bb725512ae3063af9662d245477fbe9db2fb146200f8c8'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-storage
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:01c7cb194032e5488b4d20e29a2b474a1bfa24984893e625e2544432b950f461 {'registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:01c7cb194032e5488b4d20e29a2b474a1bfa24984893e625e2544432b950f461'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/odf-csi-addons-rhel8-operator@sha256:e58afd0f236953f91cece9de950e34caac0f8ea54eb12613c5494025944d171d {'registry.redhat.io/odf4/odf-csi-addons-rhel8-operator@sha256:b838f03f967b49b4ffcdf1596fe79a0a8480cadd7a0418ec9e216a6eaa710d2d'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/cephcsi-rhel8@sha256:4295dc01c77441833ab380c1fe5069a38f239e1de732bcc23216ec8267ec5950 {'registry.redhat.io/odf4/cephcsi-rhel8@sha256:2a90498f3a7df3a79b6b691285462d6aaa1d86164b4e63326dc4a4a67bafb418'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-csi-node-driver-registrar@sha256:8d4878150d861c7343c2e9fce5f23ed8fc4ba871f8d5ca66321a72a8011d052c {'registry.redhat.io/openshift4/ose-csi-node-driver-registrar@sha256:8d4878150d861c7343c2e9fce5f23ed8fc4ba871f8d5ca66321a72a8011d052c'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-csi-external-attacher-rhel8@sha256:8ddc467b1d6d967c8f59f5eb89152694c06b51b1302975f904483ab9ea355c9a {'registry.redhat.io/openshift4/ose-csi-external-attacher-rhel8@sha256:723c89642902e7ae176317e2533acaad35709c6bbf1dc1acab515d4c91bdd4cb'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:f7c70fdbfeea3b3d890a7b30971a4a0f4164793228283e859d2479af7ce4b385 {'registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:6c6880e494dc4f8b6cb81574f323b900cdaaac9642f076abdf09e0030bade533'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:2d9c535b8d8f18e1c4e6bbcb2b18683db42f56dad853e1ac7fa4e092a95b08b8 {'registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:2a485aa16f0d43ccff84f568cc02f5e3a82be0545869b5875ee6aebf7297bfbf'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-csi-external-snapshotter-rhel8@sha256:12ea1e66431e39621e3b74f71f82ea2dcac3634f781afd72648061009e931b79 {'registry.redhat.io/openshift4/ose-csi-external-snapshotter-rhel8@sha256:12ea1e66431e39621e3b74f71f82ea2dcac3634f781afd72648061009e931b79'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/odf-csi-addons-sidecar-rhel8@sha256:4f05b3f18e4ebd505f233e15ef027f3dabcbd5c1e6d15a30326bff7c546fdb7f {'registry.redhat.io/odf4/odf-csi-addons-sidecar-rhel8@sha256:4f05b3f18e4ebd505f233e15ef027f3dabcbd5c1e6d15a30326bff7c546fdb7f'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/mcg-core-rhel8@sha256:23d10068c2bbdf23c73c4a4380e23fe7bf367192d4d3ef85a28eaa1d6d662b58 {'registry.redhat.io/odf4/mcg-core-rhel8@sha256:23d10068c2bbdf23c73c4a4380e23fe7bf367192d4d3ef85a28eaa1d6d662b58'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/rhel8/postgresql-12@sha256:f9393bef938580aa39aacf94bc56fd6f2ac515173f770c75f7fac9650eff62ba {'registry.redhat.io/rhel8/postgresql-12@sha256:2a54f31465ebcbae699b07b50cd8fc266b5467afc5974c1735aea3cf3fc27104'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/mcg-rhel8-operator@sha256:2cc7c7c15a427a3c38c3827c63f8bca45538784cd8d8a1aeb4a99f2f9db56607 {'registry.redhat.io/odf4/mcg-rhel8-operator@sha256:2858d14b582d361d55203b2f1687a8b760e1006a6594835bd58184f0a1315a4e'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/ocs-metrics-exporter-rhel8@sha256:4e0d2f1cab1d0c3c03dcb5b616e0e7fdef62c37cf619d86251cb3cd0a9b1b080 {'registry.redhat.io/odf4/ocs-metrics-exporter-rhel8@sha256:4e0d2f1cab1d0c3c03dcb5b616e0e7fdef62c37cf619d86251cb3cd0a9b1b080'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/ocs-rhel8-operator@sha256:6b0bb7e5a9a1612aefe2daeb149795f7f562757385e564af08bd209345c7f77f {'registry.redhat.io/odf4/ocs-rhel8-operator@sha256:6b0bb7e5a9a1612aefe2daeb149795f7f562757385e564af08bd209345c7f77f'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/odf-console-rhel8@sha256:58c242e76659279cacd16ea1b51a3dddaf34f62e0ef5fa8c4601fd19d7d08dbf {'registry.redhat.io/odf4/odf-console-rhel8@sha256:58c242e76659279cacd16ea1b51a3dddaf34f62e0ef5fa8c4601fd19d7d08dbf'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/odf-rhel8-operator@sha256:705c5bf3944dda6756a2d62d37863a1b7428d6d8e27732ffa3ca536348ea8a5d {'registry.redhat.io/odf4/odf-rhel8-operator@sha256:705c5bf3944dda6756a2d62d37863a1b7428d6d8e27732ffa3ca536348ea8a5d'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:7892e9da0a70b2d7e3efd98d2cb980e485f07eddff6a0dac6d6bd6c516914f3c {'registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:03c87d18494d1d1796c8729871c001057d0fa19826a672d1bc34da6609e551ba'}
>01:08:45 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/odf4/rook-ceph-rhel8-operator@sha256:00b71650abc2994e800c3466a46883110c71282bea7a42a359159349e69b5339 {'registry.redhat.io/odf4/rook-ceph-rhel8-operator@sha256:00b71650abc2994e800c3466a46883110c71282bea7a42a359159349e69b5339'}
>01:08:45 - MainThread - tests2.conftest - INFO  - human readable ocs version info written into /root/ocpcluster/ocs_version.2022-11-29T01:08:45.103805
>01:08:45 - MainThread - tests2.conftest - INFO  - PagerDuty service is not created because platform from ['openshiftdedicated', 'rosa'] is not used
>01:08:45 - MainThread - ocs_ci.utility.utils - INFO  - testrun_name: OCS4-12-Downstream-OCP4-12-BAREMETAL-IPI-1AZ-RHCOS-3M-3W
>01:08:45 - Dummy-1 - ocs_ci.utility.utils - INFO  - Executing command: oc get Pod  -A -o yaml
>01:08:45 - Dummy-2 - ocs_ci.utility.utils - INFO  - Executing command: oc get PersistentVolume  -A -o yaml
>01:08:45 - Dummy-3 - ocs_ci.utility.utils - INFO  - Executing command: oc get StorageClass  -A -o yaml
>01:08:45 - Dummy-4 - ocs_ci.utility.utils - INFO  - Executing command: oc get PersistentVolumeClaim  -A -o yaml
>01:08:45 - Dummy-5 - ocs_ci.utility.utils - INFO  - Executing command: oc get CephFileSystem  -A -o yaml
>01:08:45 - Dummy-6 - ocs_ci.utility.utils - INFO  - Executing command: oc get Namespace  -A -o yaml
>01:08:45 - Dummy-7 - ocs_ci.utility.utils - INFO  - Executing command: oc get CephBlockPool  -A -o yaml
>01:08:45 - Dummy-8 - ocs_ci.utility.utils - INFO  - Executing command: oc get volumesnapshot  -A -o yaml
>01:08:53 - MainThread - tests2.conftest - INFO  - Checking for Ceph Health OK 
>01:08:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc wait --for condition=ready pod -l app=rook-ceph-tools -n openshift-storage --timeout=300s
>01:08:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pod -l 'app=rook-ceph-tools' -o jsonpath='{.items[0].metadata.name}'
>01:08:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage exec rook-ceph-tools-749dfcffb9-jr87p -- ceph health
>01:08:54 - MainThread - ocs_ci.utility.utils - INFO  - Ceph cluster health is HEALTH_OK.
>01:08:54 - MainThread - tests2.conftest - INFO  - Ceph health check passed at setup
>01:08:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: ['oc', 'login', '-u', 'kubeadmin', '-p', '*****']
>01:08:54 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: error: x509: certificate signed by unknown authority
>
>01:08:54 - MainThread - tests2.conftest - ERROR  - There was a problem with connecting to Prometheus
>Traceback (most recent call last):
>  File "/root/ocpcluster/ocs-ci/tests2/conftest.py", line 3417, in log_alerts
>    prometheus = PrometheusAPI(threading_lock=threading_lock)
>  File "/root/ocpcluster/ocs-ci/ocs_ci/utility/prometheus.py", line 345, in __init__
>    self.refresh_connection()
>  File "/root/ocpcluster/ocs-ci/ocs_ci/utility/prometheus.py", line 363, in refresh_connection
>    login_ok = ocp.login(self._user, self._password)
>  File "/root/ocpcluster/ocs-ci/ocs_ci/ocs/ocp.py", line 493, in login
>    status = exec_cmd(
>  File "/root/ocpcluster/ocs-ci/ocs_ci/utility/utils.py", line 636, in exec_cmd
>    raise CommandFailed(
>ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: ['oc', 'login', '-u', 'kubeadmin', '-p', '*****'].
>Error is error: x509: certificate signed by unknown authority
>
>01:08:54 - MainThread - tests2.conftest - ERROR  - There was a problem with collecting alerts for analysis
>Traceback (most recent call last):
>  File "/root/ocpcluster/ocs-ci/tests2/conftest.py", line 3423, in _collect_alerts
>    alerts_response = prometheus.get(
>AttributeError: 'NoneType' object has no attribute 'get'
>01:08:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Setting up the test environment
>01:08:54 - MainThread - ocs_ci.ocs.perftests - INFO  - Setting up test environment
>01:08:54 - MainThread - ocs_ci.ocs.perftests - INFO  - Logs file path name is : /tmp/ocs-ci-logs-1669712909/tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py/TestPvcMultiSnapshotPerformance/test_pvc_multiple_snapshot_performance-CephBlockPool-512
>01:08:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: git config --get user.name
>01:08:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-console get  route -n openshift-console -o yaml
>01:08:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get  nodes -o yaml
>01:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get  clusterversion -n openshift-storage -o yaml
>01:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get  clusterversion -n openshift-storage -o yaml
>01:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>01:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>01:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:08:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph version --format json-pretty
>01:08:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>01:08:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>01:08:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p rook version
>01:09:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get csv  -n openshift-storage -o yaml
>01:09:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>01:09:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master0.ocp.openshift.lan
>01:09:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>01:09:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master1.ocp.openshift.lan
>01:09:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>01:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master2.ocp.openshift.lan
>01:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>01:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker1.ocp.openshift.lan
>01:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>01:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker2.ocp.openshift.lan
>01:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>01:09:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker3.ocp.openshift.lan
>01:09:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>01:09:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get clusterversion version -o jsonpath='{.spec.clusterID}'
>01:09:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephCluster  -n openshift-storage -o yaml
>01:09:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephFilesystem  -n openshift-storage -o yaml
>01:09:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephBlockPool  -n openshift-storage -o yaml
>01:09:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-mon -o yaml
>01:09:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-mon-a-7885556b8d-nfw4h -n openshift-storage
>01:09:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-mon-b-56696c6688-jk7m8 -n openshift-storage
>01:09:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-mon-c-655dfcd8b8-8b6wg -n openshift-storage
>01:09:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-mds -o yaml
>01:09:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-mgr -o yaml
>01:09:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-osd -o yaml
>01:09:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=noobaa -o yaml
>01:09:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-rgw -o yaml
>01:09:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>01:09:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>01:09:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:15 - MainThread - ocs_ci.ocs.cluster - INFO  - port=3300
>01:09:15 - MainThread - ocs_ci.ocs.cluster - INFO  - port=3300
>01:09:15 - MainThread - ocs_ci.ocs.cluster - INFO  - port=3300
>01:09:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephCluster ocs-storagecluster-cephcluster -n openshift-storage -o yaml
>01:09:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephFilesystem ocs-storagecluster-cephfilesystem -n openshift-storage -o yaml
>01:09:15 - MainThread - ocs_ci.ocs.cluster - INFO  - Number of mons = 3
>01:09:15 - MainThread - ocs_ci.ocs.cluster - INFO  - Number of mds = 2
>01:09:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph df --format json-pretty
>01:09:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>01:09:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>01:09:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd df --format json-pretty
>01:09:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=cluster.ocs.openshift.io/openshift-storage='' -o yaml
>01:09:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>01:09:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/master -o yaml
>01:09:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage debug nodes/smc-conv-master0.ocp.openshift.lan --to-namespace=openshift-storage  -- chroot /host /bin/bash -c "lscpu | grep '^CPU(s):' | awk '{print $NF}' || echo 'CMD FAILED'; "
>01:09:20 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
>Starting pod/smc-conv-master0ocpopenshiftlan-debug ...
>To use host binaries, run `chroot /host`
>
>Removing debug pod ...
>
>01:09:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage debug nodes/smc-conv-master0.ocp.openshift.lan --to-namespace=openshift-storage  -- chroot /host /bin/bash -c "free | grep Mem | awk '{print $2}' || echo 'CMD FAILED'; "
>01:09:21 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
>Starting pod/smc-conv-master0ocpopenshiftlan-debug ...
>To use host binaries, run `chroot /host`
>
>Removing debug pod ...
>
>01:09:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/worker -o yaml
>01:09:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage debug nodes/smc-conv-worker1.ocp.openshift.lan --to-namespace=openshift-storage  -- chroot /host /bin/bash -c "lscpu | grep '^CPU(s):' | awk '{print $NF}' || echo 'CMD FAILED'; "
>01:09:22 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
>Starting pod/smc-conv-worker1ocpopenshiftlan-debug ...
>To use host binaries, run `chroot /host`
>
>Removing debug pod ...
>
>01:09:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage debug nodes/smc-conv-worker1.ocp.openshift.lan --to-namespace=openshift-storage  -- chroot /host /bin/bash -c "free | grep Mem | awk '{print $2}' || echo 'CMD FAILED'; "
>01:09:23 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
>Starting pod/smc-conv-worker1ocpopenshiftlan-debug ...
>To use host binaries, run `chroot /host`
>
>Removing debug pod ...
>
>01:09:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>01:09:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>01:09:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph mgr dump --format json-pretty
>01:09:26 - MainThread - ocs_ci.ocs.cluster - INFO  - Number of replica : 3
>01:09:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>01:09:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>01:09:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph df --format json-pretty
>01:09:29 - MainThread - ocs_ci.ocs.perftests - INFO  - Creating new namespace (namespace-pas-test-ff8f6cdb7dc84cc398644) for the test
>01:09:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc new-project namespace-pas-test-ff8f6cdb7dc84cc398644
>01:09:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get clusterversion -n openshift-storage -o yaml
>-------------------------------- live log call ---------------------------------
>01:09:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Logs file path name is : /tmp/ocs-ci-logs-1669712909/tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py/TestPvcMultiSnapshotPerformance/test_pvc_multiple_snapshot_performance-CephBlockPool-512
>01:09:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Reslut path is : /tmp/ocs-ci-logs-1669712909/tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py/TestPvcMultiSnapshotPerformance/test_pvc_multiple_snapshot_performance
>01:09:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Going to create 512 CephBlockPool snapshots
>01:09:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage apply -f /tmp/CephBlockPoolyij1b6t_
>01:09:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephBlockPool ocs-storagecluster-cephblockpool -n openshift-storage -o yaml
>01:09:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephBlockPool pas-testing-rbd -n openshift-storage -o yaml
>01:09:31 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>01:09:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephBlockPool pas-testing-rbd -n openshift-storage -o yaml
>01:09:34 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>01:09:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephBlockPool pas-testing-rbd -n openshift-storage -o yaml
>01:09:37 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>01:09:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephBlockPool pas-testing-rbd -n openshift-storage -o yaml
>01:09:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool set pas-testing-rbd pg_num 128
>01:09:41 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: set pool 21 pg_num to 128
>
>01:09:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool set pas-testing-rbd pgp_num 128
>01:09:42 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: set pool 21 pgp_num to 128
>
>01:09:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool set pas-testing-rbd pg_num_min 128
>01:09:43 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: set pool 21 pg_num_min to 128
>
>01:09:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool set ocs-storagecluster-cephblockpool target_size_ratio 0.24
>01:09:44 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: set pool 1 target_size_ratio to 0.24
>
>01:09:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool set ocs-storagecluster-cephfilesystem-data0 target_size_ratio 0.24
>01:09:45 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: set pool 10 target_size_ratio to 0.24
>
>01:09:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>01:09:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>01:09:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>01:09:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph auth get-key client.admin --format json-pretty
>01:09:48 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding Secret with name secret-test-rbd-1cc0e6579c1f4b038d574c3a
>01:09:48 - MainThread - ocs_ci.utility.templating - INFO  - apiVersion: v1
>kind: Secret
>metadata:
>  name: secret-test-rbd-1cc0e6579c1f4b038d574c3a
>  namespace: openshift-storage
>stringData:
>  userID: admin
>  userKey: '*****'
>
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage create -f /tmp/Secretlw_giqln -o yaml
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Secret secret-test-rbd-1cc0e6579c1f4b038d574c3a -n openshift-storage -o yaml
>01:09:48 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding StorageClass with name pas-testing-rbd
>01:09:48 - MainThread - ocs_ci.utility.templating - INFO  - allowVolumeExpansion: true
>apiVersion: storage.k8s.io/v1
>kind: StorageClass
>metadata:
>  name: pas-testing-rbd
>  namespace: openshift-storage
>parameters:
>  clusterID: openshift-storage
>  csi.storage.k8s.io/controller-expand-secret-name: '*****'
>  csi.storage.k8s.io/controller-expand-secret-namespace: '*****'
>  csi.storage.k8s.io/node-stage-secret-name: '*****'
>  csi.storage.k8s.io/node-stage-secret-namespace: '*****'
>  csi.storage.k8s.io/provisioner-secret-name: '*****'
>  csi.storage.k8s.io/provisioner-secret-namespace: '*****'
>  imageFeatures: layering
>  imageFormat: '2'
>  pool: pas-testing-rbd
>provisioner: openshift-storage.rbd.csi.ceph.com
>reclaimPolicy: Delete
>volumeBindingMode: Immediate
>
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage create -f /tmp/StorageClasshdm7p5cp -o yaml
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get StorageClass pas-testing-rbd -n openshift-storage -o yaml
>01:09:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - The new SC is : pas-testing-rbd
>01:09:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Creating new snapshot class : pas-test-rbd-snapshot-class
>01:09:48 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding VolumeSnapshotClass with name pas-test-rbd-snapshot-class
>01:09:48 - MainThread - ocs_ci.utility.templating - INFO  - apiVersion: snapshot.storage.k8s.io/v1
>deletionPolicy: Delete
>driver: openshift-storage.rbd.csi.ceph.com
>kind: VolumeSnapshotClass
>metadata:
>  name: pas-test-rbd-snapshot-class
>  namespace: openshift-storage
>parameters:
>  clusterID: openshift-storage
>  csi.storage.k8s.io/snapshotter-secret-name: '*****'
>  csi.storage.k8s.io/snapshotter-secret-namespace: '*****'
>
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage create -f /tmp/VolumeSnapshotClasshbrmh2y0 -o yaml
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get VolumeSnapshotClass pas-test-rbd-snapshot-class -n openshift-storage -o yaml
>01:09:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Creating 9 GiB PVC of CephBlockPool
>01:09:48 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding PersistentVolumeClaim with name pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3
>01:09:48 - MainThread - ocs_ci.utility.templating - INFO  - apiVersion: v1
>kind: PersistentVolumeClaim
>metadata:
>  name: pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3
>  namespace: namespace-pas-test-ff8f6cdb7dc84cc398644
>spec:
>  accessModes:
>  - ReadWriteOnce
>  resources:
>    requests:
>      storage: 9Gi
>  storageClassName: pas-testing-rbd
>
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/PersistentVolumeClaimkimqfuxy -o yaml
>01:09:48 - MainThread - ocs_ci.ocs.ocp - INFO  - Waiting for a resource(s) of kind PersistentVolumeClaim identified by name 'pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3' using selector None at column name STATUS to reach desired condition Bound
>01:09:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 -n namespace-pas-test-ff8f6cdb7dc84cc398644
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get PersistentVolumeClaim  -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:49 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 at STATUS reached condition!
>01:09:49 - MainThread - ocs_ci.helpers.helpers - INFO  - PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 reached state Bound
>01:09:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Creating A POD
>01:09:49 - MainThread - ocs_ci.helpers.helpers - INFO  - Creating new Pod pod-test-rbd-2f1314eacb67426f8ceded15548 for test
>01:09:49 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding Pod with name pod-test-rbd-2f1314eacb67426f8ceded15548
>01:09:49 - MainThread - ocs_ci.utility.templating - INFO  - apiVersion: v1
>kind: Pod
>metadata:
>  name: pod-test-rbd-2f1314eacb67426f8ceded15548
>  namespace: namespace-pas-test-ff8f6cdb7dc84cc398644
>spec:
>  containers:
>  - command:
>    - /bin/sh
>    image: quay.io/ocsci/perf:latest
>    imagePullPolicy: IfNotPresent
>    name: performance
>    stdin: true
>    tty: true
>    volumeMounts:
>    - mountPath: /mnt
>      name: mypvc
>  volumes:
>  - name: mypvc
>    persistentVolumeClaim:
>      claimName: pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3
>      readOnly: false
>
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/POD_8a0r3okd -o yaml
>01:09:49 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "performance" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "performance" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "performance" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "performance" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
>
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:49 - MainThread - ocs_ci.ocs.ocp - INFO  - Waiting for a resource(s) of kind Pod identified by name 'pod-test-rbd-2f1314eacb67426f8ceded15548' using selector None at column name STATUS to reach desired condition Running
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod  -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:49 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-rbd-2f1314eacb67426f8ceded15548 at column STATUS was ContainerCreating, but we were waiting for Running
>01:09:49 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>01:09:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644
>01:09:52 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-rbd-2f1314eacb67426f8ceded15548 at column STATUS was ContainerCreating, but we were waiting for Running
>01:09:52 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>01:09:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644
>01:09:56 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-rbd-2f1314eacb67426f8ceded15548 at column STATUS was ContainerCreating, but we were waiting for Running
>01:09:56 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>01:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644
>01:09:59 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-rbd-2f1314eacb67426f8ceded15548 at STATUS reached condition!
>01:09:59 - MainThread - ocs_ci.helpers.helpers - INFO  - Pod pod-test-rbd-2f1314eacb67426f8ceded15548 reached state Running
>01:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Total capacity size is : 7173 GiB, Going to use 693 GiB, With 512 Snapshots to 9 GiB PVC. File size to be written is : 7372M with the name of pod-test-rbd-2f1314eacb67426f8ceded15548
>01:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>01:09:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - The FIO command is : fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:09:59 - MainThread - ocs_ci.ocs.perfresult - INFO  - Connecting to ES 10.0.78.167 on port 9200
>01:10:09 - MainThread - ocs_ci.ocs.perfresult - WARNING  - Cannot connect to ES server 10.0.78.167:9200
>01:10:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 1
>01:10:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:10:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 1
>01:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapr68rlnby.yaml with timeout of 600
>01:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-1-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:10:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-1-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:10:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-1-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:10:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-1-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:10:20Z with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:10:20Z with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:10:20Z with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:10:20Z with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:10:20Z with timeout of 600
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:10:20Z with timeout of 600
>01:10:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 1 are : Creation time is 1.983309 , Creation speed 3.6302966406142465, Csi creation time is 1.939227
>01:10:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 2
>01:10:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:10:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 2
>01:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjflinmqb.yaml with timeout of 600
>01:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-2-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:10:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-2-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-2-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:10:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-2-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:10:44Z with timeout of 600
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:10:44Z with timeout of 600
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:10:44Z with timeout of 600
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:10:44Z with timeout of 600
>01:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:10:44Z with timeout of 600
>01:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:10:44Z with timeout of 600
>01:10:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 2 are : Creation time is 1.356542 , Creation speed 5.307613033728407, Csi creation time is 1.322164
>01:10:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 3
>01:10:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:11:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 3
>01:11:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap27zewe82.yaml with timeout of 600
>01:11:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-3-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:11:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-3-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-3-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:11:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-3-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:11:07Z with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:11:07Z with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:11:07Z with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:11:07Z with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:11:07Z with timeout of 600
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:11:07Z with timeout of 600
>01:11:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 3 are : Creation time is 1.761387 , Creation speed 4.087687714284255, Csi creation time is 1.721786
>01:11:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 4
>01:11:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:11:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 4
>01:11:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snappa8x13g9.yaml with timeout of 600
>01:11:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-4-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:11:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-4-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:11:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-4-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:11:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-4-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:11:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:11:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:11:31Z with timeout of 600
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:11:31Z with timeout of 600
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:11:31Z with timeout of 600
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:11:31Z with timeout of 600
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:11:31Z with timeout of 600
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:11:31Z with timeout of 600
>01:11:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 4 are : Creation time is 1.994804 , Creation speed 3.6093771618665293, Csi creation time is 1.957414
>01:11:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 5
>01:11:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:11:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 5
>01:11:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapv26yj7_v.yaml with timeout of 600
>01:11:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:11:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-5-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:11:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-5-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-5-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:12:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-5-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:11:54Z with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:11:54Z with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:11:54Z with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:11:54Z with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:11:54Z with timeout of 600
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:11:54Z with timeout of 600
>01:12:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 5 are : Creation time is 1.768918 , Creation speed 4.070284772951601, Csi creation time is 1.730109
>01:12:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 6
>01:12:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:12:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 6
>01:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap207r2rmy.yaml with timeout of 600
>01:12:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-6-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:12:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-6-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-6-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:12:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-6-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:12:17Z with timeout of 600
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:12:17Z with timeout of 600
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:12:17Z with timeout of 600
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:12:17Z with timeout of 600
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:12:17Z with timeout of 600
>01:12:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:12:17Z with timeout of 600
>01:12:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 6 are : Creation time is 1.655247 , Creation speed 4.349803986957838, Csi creation time is 1.61959
>01:12:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 7
>01:12:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:12:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 7
>01:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsyz59ezs.yaml with timeout of 600
>01:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-7-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:12:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-7-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-7-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:12:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-7-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:12:41Z with timeout of 600
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:12:41Z with timeout of 600
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:12:41Z with timeout of 600
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:12:41Z with timeout of 600
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:12:41Z with timeout of 600
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:12:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:12:41Z with timeout of 600
>01:12:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 7 are : Creation time is 1.258616 , Creation speed 5.720569260203272, Csi creation time is 1.217052
>01:12:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 8
>01:12:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:13:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 8
>01:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapyq0aw8md.yaml with timeout of 600
>01:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-8-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:13:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-8-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-8-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:13:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-8-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:13:05Z with timeout of 600
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:13:05Z with timeout of 600
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:13:05Z with timeout of 600
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:13:05Z with timeout of 600
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:13:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:13:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:13:05Z with timeout of 600
>01:13:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:13:05Z with timeout of 600
>01:13:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 8 are : Creation time is 1.474281 , Creation speed 4.883736546831981, Csi creation time is 1.435087
>01:13:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 9
>01:13:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:13:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 9
>01:13:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap453m9f9p.yaml with timeout of 600
>01:13:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-9-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:13:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-9-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-9-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:13:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-9-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:13:28Z with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:13:28Z with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:13:28Z with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:13:28Z with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:13:28Z with timeout of 600
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:13:28Z with timeout of 600
>01:13:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 9 are : Creation time is 1.78569 , Creation speed 4.032054835945769, Csi creation time is 1.743977
>01:13:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 10
>01:13:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:13:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 10
>01:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaplb4lt9e5.yaml with timeout of 600
>01:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-10-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:13:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-10-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-10-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:14:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-10-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:13:52Z with timeout of 600
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:13:52Z with timeout of 600
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:13:52Z with timeout of 600
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:13:52Z with timeout of 600
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:13:52Z with timeout of 600
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:13:52Z with timeout of 600
>01:14:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 10 are : Creation time is 2.048741 , Creation speed 3.5143534492646946, Csi creation time is 2.01101
>01:14:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 11
>01:14:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:14:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 11
>01:14:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaptcv6r2l7.yaml with timeout of 600
>01:14:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-11-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:14:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-11-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-11-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:14:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-11-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:14:16Z with timeout of 600
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:14:16Z with timeout of 600
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:14:16Z with timeout of 600
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:14:16Z with timeout of 600
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:14:16Z with timeout of 600
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:14:16Z with timeout of 600
>01:14:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 11 are : Creation time is 1.384869 , Creation speed 5.199047707761529, Csi creation time is 1.343786
>01:14:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 12
>01:14:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:14:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 12
>01:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap29x45y4t.yaml with timeout of 600
>01:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-12-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:14:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-12-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-12-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:14:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-12-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:14:40Z with timeout of 600
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:14:40Z with timeout of 600
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:14:40Z with timeout of 600
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:14:40Z with timeout of 600
>01:14:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:14:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:14:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:14:40Z with timeout of 600
>01:14:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:14:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:14:40Z with timeout of 600
>01:14:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 12 are : Creation time is 1.51389 , Creation speed 4.755959812139587, Csi creation time is 1.478716
>01:14:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 13
>01:14:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:15:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 13
>01:15:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwer5of0p.yaml with timeout of 600
>01:15:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-13-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:15:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-13-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-13-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:15:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-13-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:15:04Z with timeout of 600
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:15:04Z with timeout of 600
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:15:04Z with timeout of 600
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:15:04Z with timeout of 600
>01:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:15:04Z with timeout of 600
>01:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:15:04Z with timeout of 600
>01:15:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 13 are : Creation time is 1.515832 , Creation speed 4.749866739849798, Csi creation time is 1.473843
>01:15:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 14
>01:15:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:15:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 14
>01:15:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap06aiksw8.yaml with timeout of 600
>01:15:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-14-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:15:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-14-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-14-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:15:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-14-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:15:28Z with timeout of 600
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:15:28Z with timeout of 600
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:15:28Z with timeout of 600
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:15:28Z with timeout of 600
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:15:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:15:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:15:28Z with timeout of 600
>01:15:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:15:28Z with timeout of 600
>01:15:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 14 are : Creation time is 1.629822 , Creation speed 4.417660333459727, Csi creation time is 1.5769
>01:15:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 15
>01:15:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:15:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 15
>01:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3nw4u5f2.yaml with timeout of 600
>01:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-15-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:15:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-15-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-15-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:16:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-15-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:15:52Z with timeout of 600
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:15:52Z with timeout of 600
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:15:52Z with timeout of 600
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:15:52Z with timeout of 600
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:15:52Z with timeout of 600
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:15:52Z with timeout of 600
>01:16:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 15 are : Creation time is 1.226132 , Creation speed 5.87212469783025, Csi creation time is 1.183291
>01:16:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 16
>01:16:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:16:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 16
>01:16:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapi_reypa9.yaml with timeout of 600
>01:16:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-16-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:16:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-16-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-16-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:16:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-16-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:16:16Z with timeout of 600
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:16:16Z with timeout of 600
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:16:16Z with timeout of 600
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:16:16Z with timeout of 600
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:16:16Z with timeout of 600
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:16:16Z with timeout of 600
>01:16:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 16 are : Creation time is 1.417345 , Creation speed 5.079920555686865, Csi creation time is 1.382469
>01:16:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 17
>01:16:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:16:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 17
>01:16:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwvcq7bk9.yaml with timeout of 600
>01:16:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-17-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:16:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-17-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-17-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:16:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-17-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:16:39Z with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:16:39Z with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:16:39Z with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:16:39Z with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:16:39Z with timeout of 600
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:16:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:16:39Z with timeout of 600
>01:16:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 17 are : Creation time is 2.059245 , Creation speed 3.4964270885688684, Csi creation time is 2.019557
>01:16:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 18
>01:16:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:17:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 18
>01:17:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbgz4nkqu.yaml with timeout of 600
>01:17:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-18-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:17:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-18-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-18-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:17:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-18-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:17:05Z with timeout of 600
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:17:05Z with timeout of 600
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:17:05Z with timeout of 600
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:17:05Z with timeout of 600
>01:17:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:17:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:17:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:17:05Z with timeout of 600
>01:17:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:17:05Z with timeout of 600
>01:17:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 18 are : Creation time is 1.495255 , Creation speed 4.815232184476895, Csi creation time is 1.459228
>01:17:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 19
>01:17:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:17:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 19
>01:17:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapw1uy959x.yaml with timeout of 600
>01:17:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-19-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:17:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-19-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-19-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:17:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-19-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:17:29Z with timeout of 600
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:17:29Z with timeout of 600
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:17:29Z with timeout of 600
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:17:29Z with timeout of 600
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:17:29Z with timeout of 600
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:17:29Z with timeout of 600
>01:17:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 19 are : Creation time is 1.401208 , Creation speed 5.138423417508322, Csi creation time is 1.359866
>01:17:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 20
>01:17:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:17:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 20
>01:17:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapoz63sdh4.yaml with timeout of 600
>01:17:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:17:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-20-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:17:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-20-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-20-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:18:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-20-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:17:53Z with timeout of 600
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:17:53Z with timeout of 600
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:17:53Z with timeout of 600
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:17:53Z with timeout of 600
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:17:53Z with timeout of 600
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:17:53Z with timeout of 600
>01:18:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 20 are : Creation time is 1.195491 , Creation speed 6.022630032346542, Csi creation time is 1.158963
>01:18:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 21
>01:18:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:18:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 21
>01:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_qqpuvfv.yaml with timeout of 600
>01:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-21-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:18:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-21-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-21-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:18:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-21-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:18:17Z with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:18:17Z with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:18:17Z with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:18:17Z with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:18:17Z with timeout of 600
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:18:17Z with timeout of 600
>01:18:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 21 are : Creation time is 1.905634 , Creation speed 3.7782701190260037, Csi creation time is 1.864854
>01:18:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 22
>01:18:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:18:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 22
>01:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapeu5udt3l.yaml with timeout of 600
>01:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-22-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:18:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-22-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-22-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:18:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-22-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:18:41Z with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:18:41Z with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:18:41Z with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:18:41Z with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:18:41Z with timeout of 600
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:18:41Z with timeout of 600
>01:18:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 22 are : Creation time is 1.832156 , Creation speed 3.9297963710513737, Csi creation time is 1.792483
>01:18:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 23
>01:18:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:19:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 23
>01:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0q89z073.yaml with timeout of 600
>01:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-23-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:19:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-23-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-23-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:19:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-23-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:19:06Z with timeout of 600
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:19:06Z with timeout of 600
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:19:06Z with timeout of 600
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:19:06Z with timeout of 600
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:19:06Z with timeout of 600
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:19:06Z with timeout of 600
>01:19:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 23 are : Creation time is 2.011938 , Creation speed 3.578639103193041, Csi creation time is 1.973634
>01:19:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 24
>01:19:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:19:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 24
>01:19:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwp5gp1lu.yaml with timeout of 600
>01:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-24-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:19:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-24-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-24-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:19:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-24-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:19:31Z with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:19:31Z with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:19:31Z with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:19:31Z with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:19:31Z with timeout of 600
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:19:31Z with timeout of 600
>01:19:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 24 are : Creation time is 1.861759 , Creation speed 3.8673104306196455, Csi creation time is 1.825476
>01:19:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 25
>01:19:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:19:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 25
>01:19:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapnhujvmhr.yaml with timeout of 600
>01:19:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:19:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-25-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:19:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-25-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:20:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-25-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:20:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-25-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:19:56Z with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:19:56Z with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:19:56Z with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:19:56Z with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:19:56Z with timeout of 600
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:19:56Z with timeout of 600
>01:20:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 25 are : Creation time is 2.054402 , Creation speed 3.504669485329551, Csi creation time is 2.014777
>01:20:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 26
>01:20:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:20:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 26
>01:20:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap21fgrl7j.yaml with timeout of 600
>01:20:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-26-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:20:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-26-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:20:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-26-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:20:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-26-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:20:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:20:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:20:20Z with timeout of 600
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:20:20Z with timeout of 600
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:20:20Z with timeout of 600
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:20:20Z with timeout of 600
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:20:20Z with timeout of 600
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:20:20Z with timeout of 600
>01:20:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 26 are : Creation time is 1.179675 , Creation speed 6.103375929811177, Csi creation time is 1.136756
>01:20:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 27
>01:20:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:20:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 27
>01:20:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaptfwnow9b.yaml with timeout of 600
>01:20:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-27-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:20:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-27-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:20:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-27-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:20:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-27-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:20:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:20:44Z with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:20:44Z with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:20:44Z with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:20:44Z with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:20:44Z with timeout of 600
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:20:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:20:44Z with timeout of 600
>01:20:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 27 are : Creation time is 1.151656 , Creation speed 6.251866876914634, Csi creation time is 1.111642
>01:20:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 28
>01:20:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:21:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 28
>01:21:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdrukd9st.yaml with timeout of 600
>01:21:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-28-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:21:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-28-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-28-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:21:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-28-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:21:08Z with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:21:08Z with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:21:08Z with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:21:08Z with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:21:08Z with timeout of 600
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:21:08Z with timeout of 600
>01:21:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 28 are : Creation time is 1.947051 , Creation speed 3.6979000550062633, Csi creation time is 1.905934
>01:21:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 29
>01:21:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:21:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 29
>01:21:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapzglnuigq.yaml with timeout of 600
>01:21:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-29-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:21:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-29-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:21:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-29-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:21:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-29-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:21:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:21:32Z with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:21:32Z with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:21:32Z with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:21:32Z with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:21:32Z with timeout of 600
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:21:32Z with timeout of 600
>01:21:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 29 are : Creation time is 1.915654 , Creation speed 3.7585075384176894, Csi creation time is 1.875206
>01:21:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 30
>01:21:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:21:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 30
>01:21:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapn44bnyfi.yaml with timeout of 600
>01:21:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:21:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-30-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:21:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-30-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-30-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:22:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-30-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:21:58Z with timeout of 600
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:21:58Z with timeout of 600
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:21:58Z with timeout of 600
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:21:58Z with timeout of 600
>01:22:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:22:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:22:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:21:58Z with timeout of 600
>01:22:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:21:58Z with timeout of 600
>01:22:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 30 are : Creation time is 1.583345 , Creation speed 4.547334914374315, Csi creation time is 1.545339
>01:22:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 31
>01:22:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:22:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 31
>01:22:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapflcggqgv.yaml with timeout of 600
>01:22:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-31-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:22:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-31-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-31-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:22:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-31-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:22:22Z with timeout of 600
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:22:22Z with timeout of 600
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:22:22Z with timeout of 600
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:22:22Z with timeout of 600
>01:22:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:22:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:22:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:22:22Z with timeout of 600
>01:22:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:22:22Z with timeout of 600
>01:22:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 31 are : Creation time is 1.663235 , Creation speed 4.328913232345399, Csi creation time is 1.625771
>01:22:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 32
>01:22:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:22:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 32
>01:22:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapk8uuscgx.yaml with timeout of 600
>01:22:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-32-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:22:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-32-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-32-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:22:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-32-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:22:47Z with timeout of 600
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:22:47Z with timeout of 600
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:22:47Z with timeout of 600
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:22:47Z with timeout of 600
>01:22:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:22:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:22:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:22:47Z with timeout of 600
>01:22:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:22:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:22:47Z with timeout of 600
>01:22:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 32 are : Creation time is 1.656779 , Creation speed 4.345781785017797, Csi creation time is 1.61829
>01:22:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 33
>01:22:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:23:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 33
>01:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfj0rvsm9.yaml with timeout of 600
>01:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-33-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:23:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-33-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:23:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-33-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:23:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-33-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:23:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:23:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:23:12Z with timeout of 600
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:23:12Z with timeout of 600
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:23:12Z with timeout of 600
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:23:12Z with timeout of 600
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:23:12Z with timeout of 600
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:23:12Z with timeout of 600
>01:23:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 33 are : Creation time is 2.030174 , Creation speed 3.5464940443528485, Csi creation time is 1.990477
>01:23:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 34
>01:23:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:23:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 34
>01:23:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapl61_wrze.yaml with timeout of 600
>01:23:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-34-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:23:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-34-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-34-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:23:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-34-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:23:37Z with timeout of 600
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:23:37Z with timeout of 600
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:23:37Z with timeout of 600
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:23:37Z with timeout of 600
>01:23:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:23:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:23:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:23:37Z with timeout of 600
>01:23:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:23:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:23:37Z with timeout of 600
>01:23:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 34 are : Creation time is 1.443131 , Creation speed 4.98915205896069, Csi creation time is 1.405375
>01:23:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 35
>01:23:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:24:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 35
>01:24:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdf1xxfyb.yaml with timeout of 600
>01:24:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-35-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:24:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-35-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:24:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-35-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:24:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-35-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:24:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:24:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:24:01Z with timeout of 600
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:24:01Z with timeout of 600
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:24:01Z with timeout of 600
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:24:01Z with timeout of 600
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:24:01Z with timeout of 600
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:24:01Z with timeout of 600
>01:24:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 35 are : Creation time is 2.038452 , Creation speed 3.53209199922294, Csi creation time is 1.990915
>01:24:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 36
>01:24:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:24:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 36
>01:24:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snape0pvvuwr.yaml with timeout of 600
>01:24:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-36-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:24:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-36-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:24:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-36-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:24:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-36-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:24:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:24:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:24:25Z with timeout of 600
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:24:25Z with timeout of 600
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:24:25Z with timeout of 600
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:24:25Z with timeout of 600
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:24:25Z with timeout of 600
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:24:25Z with timeout of 600
>01:24:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 36 are : Creation time is 1.278817 , Creation speed 5.630203539677686, Csi creation time is 1.242624
>01:24:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 37
>01:24:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:24:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 37
>01:24:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap10pyj43b.yaml with timeout of 600
>01:24:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-37-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:24:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-37-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-37-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:24:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-37-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:24:49Z with timeout of 600
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:24:49Z with timeout of 600
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:24:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:24:49Z with timeout of 600
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:24:49Z with timeout of 600
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:24:49Z with timeout of 600
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:24:49Z with timeout of 600
>01:25:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 37 are : Creation time is 1.465267 , Creation speed 4.913780218895259, Csi creation time is 1.423035
>01:25:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 38
>01:25:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:25:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 38
>01:25:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapg01_6sxo.yaml with timeout of 600
>01:25:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-38-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:25:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-38-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-38-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:25:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-38-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:25:13Z with timeout of 600
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:25:13Z with timeout of 600
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:25:13Z with timeout of 600
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:25:13Z with timeout of 600
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:25:13Z with timeout of 600
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:25:13Z with timeout of 600
>01:25:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 38 are : Creation time is 1.375405 , Creation speed 5.234821743413758, Csi creation time is 1.339027
>01:25:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 39
>01:25:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:25:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 39
>01:25:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapy_nrxslb.yaml with timeout of 600
>01:25:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-39-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:25:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-39-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-39-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:25:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-39-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:25:37Z with timeout of 600
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:25:37Z with timeout of 600
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:25:37Z with timeout of 600
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:25:37Z with timeout of 600
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:25:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:25:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:25:37Z with timeout of 600
>01:25:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:25:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:25:37Z with timeout of 600
>01:25:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 39 are : Creation time is 1.850764 , Creation speed 3.8902853092020377, Csi creation time is 1.809679
>01:25:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 40
>01:25:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:26:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 40
>01:26:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapel8ailfl.yaml with timeout of 600
>01:26:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-40-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:26:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-40-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-40-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:26:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-40-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:26:00Z with timeout of 600
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:26:00Z with timeout of 600
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:26:00Z with timeout of 600
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:26:00Z with timeout of 600
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:26:00Z with timeout of 600
>01:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:26:00Z with timeout of 600
>01:26:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 40 are : Creation time is 1.983656 , Creation speed 3.629661594550668, Csi creation time is 1.946686
>01:26:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 41
>01:26:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:26:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 41
>01:26:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjnjo65jx.yaml with timeout of 600
>01:26:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-41-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:26:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-41-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-41-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:26:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-41-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:26:24Z with timeout of 600
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:26:24Z with timeout of 600
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:26:24Z with timeout of 600
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:26:24Z with timeout of 600
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:26:24Z with timeout of 600
>01:26:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:26:24Z with timeout of 600
>01:26:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 41 are : Creation time is 1.954852 , Creation speed 3.6831432763196394, Csi creation time is 1.911774
>01:26:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 42
>01:26:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:26:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 42
>01:26:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3ga0t39q.yaml with timeout of 600
>01:26:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-42-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:26:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-42-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-42-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:26:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-42-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:26:48Z with timeout of 600
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:26:48Z with timeout of 600
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:26:48Z with timeout of 600
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:26:48Z with timeout of 600
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:26:48Z with timeout of 600
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:26:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:26:48Z with timeout of 600
>01:26:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 42 are : Creation time is 1.491758 , Creation speed 4.826520119215047, Csi creation time is 1.454643
>01:26:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 43
>01:26:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:27:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 43
>01:27:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapik0tzh_2.yaml with timeout of 600
>01:27:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-43-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:27:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-43-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:27:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-43-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:27:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-43-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:27:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:27:13Z with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:27:13Z with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:27:13Z with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:27:13Z with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:27:13Z with timeout of 600
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:27:13Z with timeout of 600
>01:27:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 43 are : Creation time is 1.161436 , Creation speed 6.199222341997321, Csi creation time is 1.118481
>01:27:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 44
>01:27:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:27:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 44
>01:27:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapj25gat8q.yaml with timeout of 600
>01:27:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-44-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:27:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-44-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-44-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:27:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-44-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:27:38Z with timeout of 600
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:27:38Z with timeout of 600
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:27:38Z with timeout of 600
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:27:38Z with timeout of 600
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:27:38Z with timeout of 600
>01:27:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:27:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:27:38Z with timeout of 600
>01:27:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 44 are : Creation time is 2.010985 , Creation speed 3.5803350099578073, Csi creation time is 1.971652
>01:27:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 45
>01:27:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:28:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 45
>01:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapezvn0qto.yaml with timeout of 600
>01:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-45-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:28:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-45-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-45-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:28:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-45-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:28:02Z with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:28:02Z with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:28:02Z with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:28:02Z with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:28:02Z with timeout of 600
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:28:02Z with timeout of 600
>01:28:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 45 are : Creation time is 1.838086 , Creation speed 3.9171181326662623, Csi creation time is 1.797608
>01:28:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 46
>01:28:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:28:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 46
>01:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapcycvqw7a.yaml with timeout of 600
>01:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-46-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:28:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-46-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-46-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:28:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-46-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:28:27Z with timeout of 600
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:28:27Z with timeout of 600
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:28:27Z with timeout of 600
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:28:27Z with timeout of 600
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:28:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:28:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:28:27Z with timeout of 600
>01:28:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:28:27Z with timeout of 600
>01:28:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 46 are : Creation time is 1.841709 , Creation speed 3.909412399027208, Csi creation time is 1.805842
>01:28:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 47
>01:28:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:28:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 47
>01:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapn6tqrekn.yaml with timeout of 600
>01:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-47-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:28:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-47-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-47-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:29:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-47-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:28:52Z with timeout of 600
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:28:52Z with timeout of 600
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:28:52Z with timeout of 600
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:28:52Z with timeout of 600
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:29:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:29:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:28:52Z with timeout of 600
>01:29:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:28:52Z with timeout of 600
>01:29:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 47 are : Creation time is 1.569931 , Creation speed 4.586188819763416, Csi creation time is 1.527698
>01:29:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 48
>01:29:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:29:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 48
>01:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmb_fhdxg.yaml with timeout of 600
>01:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-48-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:29:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-48-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-48-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:29:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-48-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:29:16Z with timeout of 600
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:29:16Z with timeout of 600
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:29:16Z with timeout of 600
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:29:16Z with timeout of 600
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:29:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:29:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:29:16Z with timeout of 600
>01:29:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:29:16Z with timeout of 600
>01:29:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 48 are : Creation time is 1.919325 , Creation speed 3.7513188230237193, Csi creation time is 1.886809
>01:29:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 49
>01:29:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:29:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 49
>01:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4j9jkbyx.yaml with timeout of 600
>01:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-49-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:29:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-49-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-49-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:29:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-49-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:29:40Z with timeout of 600
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:29:40Z with timeout of 600
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:29:40Z with timeout of 600
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:29:40Z with timeout of 600
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:29:40Z with timeout of 600
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:29:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:29:40Z with timeout of 600
>01:29:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 49 are : Creation time is 1.731083 , Creation speed 4.1592459749185915, Csi creation time is 1.692603
>01:29:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 50
>01:29:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:30:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 50
>01:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapuk_8z7vf.yaml with timeout of 600
>01:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-50-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:30:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-50-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-50-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:30:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-50-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:30:04Z with timeout of 600
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:30:04Z with timeout of 600
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:30:04Z with timeout of 600
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:30:04Z with timeout of 600
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:30:04Z with timeout of 600
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:30:04Z with timeout of 600
>01:30:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 50 are : Creation time is 1.689362 , Creation speed 4.261963984036577, Csi creation time is 1.652369
>01:30:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 51
>01:30:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:30:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 51
>01:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaps1mvunjk.yaml with timeout of 600
>01:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-51-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:30:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-51-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-51-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:30:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-51-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:30:28Z with timeout of 600
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:30:28Z with timeout of 600
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:30:28Z with timeout of 600
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:30:28Z with timeout of 600
>01:30:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:30:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:30:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:30:28Z with timeout of 600
>01:30:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:30:28Z with timeout of 600
>01:30:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 51 are : Creation time is 1.80398 , Creation speed 3.9911750684597394, Csi creation time is 1.761746
>01:30:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 52
>01:30:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:30:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 52
>01:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbq6ghpy3.yaml with timeout of 600
>01:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-52-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:30:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-52-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-52-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:31:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-52-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:30:51Z with timeout of 600
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:30:51Z with timeout of 600
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:30:51Z with timeout of 600
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:30:51Z with timeout of 600
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:30:51Z with timeout of 600
>01:31:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:30:51Z with timeout of 600
>01:31:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 52 are : Creation time is 1.804943 , Creation speed 3.9890456374522634, Csi creation time is 1.76673
>01:31:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 53
>01:31:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:31:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 53
>01:31:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap75q4eque.yaml with timeout of 600
>01:31:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-53-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:31:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-53-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-53-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:31:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-53-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:31:15Z with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:31:15Z with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:31:15Z with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:31:15Z with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:31:15Z with timeout of 600
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:31:15Z with timeout of 600
>01:31:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 53 are : Creation time is 1.161779 , Creation speed 6.197392102973113, Csi creation time is 1.124401
>01:31:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 54
>01:31:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:31:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 54
>01:31:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapu50iyum4.yaml with timeout of 600
>01:31:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-54-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:31:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-54-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-54-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:31:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-54-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:31:39Z with timeout of 600
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:31:39Z with timeout of 600
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:31:39Z with timeout of 600
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:31:39Z with timeout of 600
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:31:39Z with timeout of 600
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:31:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:31:39Z with timeout of 600
>01:31:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 54 are : Creation time is 1.597177 , Creation speed 4.507953720846218, Csi creation time is 1.559704
>01:31:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 55
>01:31:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:32:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 55
>01:32:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapxo6siiu9.yaml with timeout of 600
>01:32:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-55-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:32:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-55-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-55-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:32:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-55-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:32:03Z with timeout of 600
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:32:03Z with timeout of 600
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:32:03Z with timeout of 600
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:32:03Z with timeout of 600
>01:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:32:03Z with timeout of 600
>01:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:32:03Z with timeout of 600
>01:32:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 55 are : Creation time is 1.962937 , Creation speed 3.667973042435901, Csi creation time is 1.923437
>01:32:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 56
>01:32:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:32:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 56
>01:32:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap6nmse9t4.yaml with timeout of 600
>01:32:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-56-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:32:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-56-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-56-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:32:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-56-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:32:26Z with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:32:26Z with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:32:26Z with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:32:26Z with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:32:26Z with timeout of 600
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:32:26Z with timeout of 600
>01:32:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 56 are : Creation time is 1.855287 , Creation speed 3.880801191405966, Csi creation time is 1.818678
>01:32:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 57
>01:32:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:32:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 57
>01:32:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8fxhgool.yaml with timeout of 600
>01:32:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:32:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-57-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:32:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-57-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-57-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:33:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-57-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:32:53Z with timeout of 600
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:32:53Z with timeout of 600
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:32:53Z with timeout of 600
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:32:53Z with timeout of 600
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:32:53Z with timeout of 600
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:32:53Z with timeout of 600
>01:33:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 57 are : Creation time is 1.588089 , Creation speed 4.533750942170117, Csi creation time is 1.547868
>01:33:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 58
>01:33:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:33:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 58
>01:33:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapb3ld67uf.yaml with timeout of 600
>01:33:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-58-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:33:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-58-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:33:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-58-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:33:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-58-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:33:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:33:19Z with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:33:19Z with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:33:19Z with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:33:19Z with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:33:19Z with timeout of 600
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:33:19Z with timeout of 600
>01:33:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 58 are : Creation time is 1.447774 , Creation speed 4.973151886965784, Csi creation time is 1.411768
>01:33:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 59
>01:33:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:33:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 59
>01:33:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snappfnmszjc.yaml with timeout of 600
>01:33:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-59-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:33:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-59-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:33:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-59-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:33:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-59-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:33:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:33:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:33:44Z with timeout of 600
>01:33:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:33:44Z with timeout of 600
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:33:44Z with timeout of 600
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:33:44Z with timeout of 600
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:33:44Z with timeout of 600
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:33:44Z with timeout of 600
>01:33:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 59 are : Creation time is 1.673468 , Creation speed 4.302442592269467, Csi creation time is 1.629671
>01:33:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 60
>01:33:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:34:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 60
>01:34:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapuympjllv.yaml with timeout of 600
>01:34:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-60-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:34:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-60-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:34:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-60-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:34:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-60-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:34:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:34:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:34:08Z with timeout of 600
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:34:08Z with timeout of 600
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:34:08Z with timeout of 600
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:34:08Z with timeout of 600
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:34:08Z with timeout of 600
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:34:08Z with timeout of 600
>01:34:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 60 are : Creation time is 1.680348 , Creation speed 4.2848267144662895, Csi creation time is 1.647206
>01:34:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 61
>01:34:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:34:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 61
>01:34:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapk40ju2fp.yaml with timeout of 600
>01:34:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-61-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:34:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-61-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:34:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-61-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:34:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-61-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:34:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:34:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:34:32Z with timeout of 600
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:34:32Z with timeout of 600
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:34:32Z with timeout of 600
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:34:32Z with timeout of 600
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:34:32Z with timeout of 600
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:34:32Z with timeout of 600
>01:34:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 61 are : Creation time is 1.701259 , Creation speed 4.232159829867174, Csi creation time is 1.656604
>01:34:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 62
>01:34:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:34:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 62
>01:34:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdxwak_va.yaml with timeout of 600
>01:34:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:34:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-62-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:34:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-62-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-62-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:35:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-62-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:34:56Z with timeout of 600
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:34:56Z with timeout of 600
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:34:56Z with timeout of 600
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:34:56Z with timeout of 600
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:34:56Z with timeout of 600
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:34:56Z with timeout of 600
>01:35:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 62 are : Creation time is 1.916235 , Creation speed 3.7573679637413995, Csi creation time is 1.880625
>01:35:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 63
>01:35:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:35:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 63
>01:35:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap65xzcyp_.yaml with timeout of 600
>01:35:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-63-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:35:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-63-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-63-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:35:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-63-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:35:20Z with timeout of 600
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:35:20Z with timeout of 600
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:35:20Z with timeout of 600
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:35:20Z with timeout of 600
>01:35:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:35:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:35:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:35:20Z with timeout of 600
>01:35:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:35:20Z with timeout of 600
>01:35:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 63 are : Creation time is 1.203034 , Creation speed 5.984868258087469, Csi creation time is 1.165047
>01:35:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 64
>01:35:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:35:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 64
>01:35:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapocpi5m2c.yaml with timeout of 600
>01:35:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-64-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:35:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-64-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-64-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:35:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-64-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:35:44Z with timeout of 600
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:35:44Z with timeout of 600
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:35:44Z with timeout of 600
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:35:44Z with timeout of 600
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:35:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:35:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:35:44Z with timeout of 600
>01:35:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:35:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:35:44Z with timeout of 600
>01:35:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 64 are : Creation time is 1.999251 , Creation speed 3.6013487050900563, Csi creation time is 1.962002
>01:35:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 65
>01:35:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:36:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 65
>01:36:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8boe_e3u.yaml with timeout of 600
>01:36:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-65-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:36:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-65-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-65-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:36:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-65-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:36:07Z with timeout of 600
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:36:07Z with timeout of 600
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:36:07Z with timeout of 600
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:36:07Z with timeout of 600
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:36:07Z with timeout of 600
>01:36:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:36:07Z with timeout of 600
>01:36:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 65 are : Creation time is 1.232801 , Creation speed 5.840358662914777, Csi creation time is 1.192288
>01:36:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 66
>01:36:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:36:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 66
>01:36:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapt__xlq3e.yaml with timeout of 600
>01:36:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-66-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:36:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-66-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-66-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:36:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-66-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:36:32Z with timeout of 600
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:36:32Z with timeout of 600
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:36:32Z with timeout of 600
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:36:32Z with timeout of 600
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:36:32Z with timeout of 600
>01:36:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:36:32Z with timeout of 600
>01:36:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 66 are : Creation time is 1.754367 , Creation speed 4.104044364719583, Csi creation time is 1.719142
>01:36:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 67
>01:36:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:36:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 67
>01:36:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap__owj4qu.yaml with timeout of 600
>01:36:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:36:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-67-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:36:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-67-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-67-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:37:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-67-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:36:55Z with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:36:55Z with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:36:55Z with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:36:55Z with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:36:55Z with timeout of 600
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:36:55Z with timeout of 600
>01:37:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 67 are : Creation time is 1.285273 , Creation speed 5.601922704359307, Csi creation time is 1.24162
>01:37:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 68
>01:37:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:37:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 68
>01:37:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapn9b34g7i.yaml with timeout of 600
>01:37:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-68-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:37:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-68-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-68-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:37:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-68-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:37:21Z with timeout of 600
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:37:21Z with timeout of 600
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:37:21Z with timeout of 600
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:37:21Z with timeout of 600
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:37:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:37:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:37:21Z with timeout of 600
>01:37:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:37:21Z with timeout of 600
>01:37:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 68 are : Creation time is 1.15937 , Creation speed 6.2102693704339424, Csi creation time is 1.118564
>01:37:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 69
>01:37:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:37:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 69
>01:37:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapiasadxws.yaml with timeout of 600
>01:37:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-69-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:37:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-69-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-69-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:37:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-69-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:37:45Z with timeout of 600
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:37:45Z with timeout of 600
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:37:45Z with timeout of 600
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:37:45Z with timeout of 600
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:37:45Z with timeout of 600
>01:37:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:37:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:37:45Z with timeout of 600
>01:37:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 69 are : Creation time is 1.326112 , Creation speed 5.4294056610603025, Csi creation time is 1.28183
>01:37:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 70
>01:37:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:38:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 70
>01:38:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snape309rtwy.yaml with timeout of 600
>01:38:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-70-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:38:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-70-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-70-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:38:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-70-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:38:10Z with timeout of 600
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:38:10Z with timeout of 600
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:38:10Z with timeout of 600
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:38:10Z with timeout of 600
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:38:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:38:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:38:10Z with timeout of 600
>01:38:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:38:10Z with timeout of 600
>01:38:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 70 are : Creation time is 1.187831 , Creation speed 6.0614683401931755, Csi creation time is 1.150243
>01:38:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 71
>01:38:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:38:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 71
>01:38:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapl_a2uhga.yaml with timeout of 600
>01:38:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-71-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:38:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-71-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-71-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:38:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-71-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:38:34Z with timeout of 600
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:38:34Z with timeout of 600
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:38:34Z with timeout of 600
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:38:34Z with timeout of 600
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:38:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:38:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:38:34Z with timeout of 600
>01:38:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:38:34Z with timeout of 600
>01:38:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 71 are : Creation time is 1.313989 , Creation speed 5.479497925781723, Csi creation time is 1.270419
>01:38:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 72
>01:38:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:38:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 72
>01:38:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbqn5j52b.yaml with timeout of 600
>01:38:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:38:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-72-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:38:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-72-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-72-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:39:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-72-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:38:58Z with timeout of 600
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:38:58Z with timeout of 600
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:38:58Z with timeout of 600
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:38:58Z with timeout of 600
>01:39:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:39:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:39:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:38:58Z with timeout of 600
>01:39:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:38:58Z with timeout of 600
>01:39:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 72 are : Creation time is 1.23977 , Creation speed 5.807528815828743, Csi creation time is 1.202457
>01:39:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 73
>01:39:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:39:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 73
>01:39:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3y72rkvf.yaml with timeout of 600
>01:39:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-73-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:39:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-73-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:39:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-73-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:39:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-73-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:39:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:39:21Z with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:39:21Z with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:39:21Z with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:39:21Z with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:39:21Z with timeout of 600
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:39:21Z with timeout of 600
>01:39:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 73 are : Creation time is 1.52865 , Creation speed 4.710038269060936, Csi creation time is 1.493592
>01:39:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 74
>01:39:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:39:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 74
>01:39:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapb7ew24tw.yaml with timeout of 600
>01:39:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-74-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:39:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-74-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-74-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:39:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-74-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:39:45Z with timeout of 600
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:39:45Z with timeout of 600
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:39:45Z with timeout of 600
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:39:45Z with timeout of 600
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:39:45Z with timeout of 600
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:39:45Z with timeout of 600
>01:39:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 74 are : Creation time is 1.763571 , Creation speed 4.082625536482512, Csi creation time is 1.725693
>01:39:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 75
>01:39:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:40:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 75
>01:40:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7mdvw8ko.yaml with timeout of 600
>01:40:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-75-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:40:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-75-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-75-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:40:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-75-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:40:09Z with timeout of 600
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:40:09Z with timeout of 600
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:40:09Z with timeout of 600
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:40:09Z with timeout of 600
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:40:09Z with timeout of 600
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:40:09Z with timeout of 600
>01:40:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 75 are : Creation time is 1.920528 , Creation speed 3.7489690335157833, Csi creation time is 1.878817
>01:40:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 76
>01:40:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:40:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 76
>01:40:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1ug1qw8c.yaml with timeout of 600
>01:40:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-76-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:40:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-76-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-76-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:40:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-76-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:40:32Z with timeout of 600
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:40:32Z with timeout of 600
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:40:32Z with timeout of 600
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:40:32Z with timeout of 600
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:40:32Z with timeout of 600
>01:40:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:40:32Z with timeout of 600
>01:40:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 76 are : Creation time is 1.896529 , Creation speed 3.796409124247507, Csi creation time is 1.851749
>01:40:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 77
>01:40:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:40:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 77
>01:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapa_upkzp2.yaml with timeout of 600
>01:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-77-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:40:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-77-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-77-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:41:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-77-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:40:57Z with timeout of 600
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:40:57Z with timeout of 600
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:40:57Z with timeout of 600
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:40:57Z with timeout of 600
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:40:57Z with timeout of 600
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:40:57Z with timeout of 600
>01:41:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 77 are : Creation time is 1.843679 , Creation speed 3.905235130410446, Csi creation time is 1.803483
>01:41:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 78
>01:41:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:41:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 78
>01:41:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapu2naekoq.yaml with timeout of 600
>01:41:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-78-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:41:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-78-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-78-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:41:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-78-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:41:20Z with timeout of 600
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:41:20Z with timeout of 600
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:41:20Z with timeout of 600
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:41:20Z with timeout of 600
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:41:20Z with timeout of 600
>01:41:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:41:20Z with timeout of 600
>01:41:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 78 are : Creation time is 1.345164 , Creation speed 5.352507203582611, Csi creation time is 1.301728
>01:41:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 79
>01:41:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:41:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 79
>01:41:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapq2vdknee.yaml with timeout of 600
>01:41:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-79-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:41:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-79-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-79-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:41:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-79-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:41:44Z with timeout of 600
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:41:44Z with timeout of 600
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:41:44Z with timeout of 600
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:41:44Z with timeout of 600
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:41:44Z with timeout of 600
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:41:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:41:44Z with timeout of 600
>01:41:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 79 are : Creation time is 1.904178 , Creation speed 3.781159114326497, Csi creation time is 1.862578
>01:41:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 80
>01:41:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:42:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 80
>01:42:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snappikae15n.yaml with timeout of 600
>01:42:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-80-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:42:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-80-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-80-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:42:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-80-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:42:08Z with timeout of 600
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:42:08Z with timeout of 600
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:42:08Z with timeout of 600
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:42:08Z with timeout of 600
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:42:08Z with timeout of 600
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:42:08Z with timeout of 600
>01:42:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 80 are : Creation time is 2.117274 , Creation speed 3.4005990722032196, Csi creation time is 2.076041
>01:42:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 81
>01:42:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:42:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 81
>01:42:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapttqhs9pl.yaml with timeout of 600
>01:42:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-81-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:42:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-81-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-81-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:42:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-81-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:42:31Z with timeout of 600
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:42:31Z with timeout of 600
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:42:31Z with timeout of 600
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:42:31Z with timeout of 600
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:42:31Z with timeout of 600
>01:42:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:42:31Z with timeout of 600
>01:42:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 81 are : Creation time is 1.837545 , Creation speed 3.9182713892720997, Csi creation time is 1.79761
>01:42:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 82
>01:42:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:42:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 82
>01:42:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapws_h08dl.yaml with timeout of 600
>01:42:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:42:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-82-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:42:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-82-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-82-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:43:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-82-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:42:55Z with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:42:55Z with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:42:55Z with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:42:55Z with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:42:55Z with timeout of 600
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:42:55Z with timeout of 600
>01:43:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 82 are : Creation time is 1.405838 , Creation speed 5.121500485831227, Csi creation time is 1.367396
>01:43:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 83
>01:43:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:43:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 83
>01:43:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfzxwc0w7.yaml with timeout of 600
>01:43:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-83-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:43:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-83-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:43:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-83-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:43:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-83-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:43:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:43:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:43:19Z with timeout of 600
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:43:19Z with timeout of 600
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:43:19Z with timeout of 600
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:43:19Z with timeout of 600
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:43:19Z with timeout of 600
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:43:19Z with timeout of 600
>01:43:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 83 are : Creation time is 1.753719 , Creation speed 4.105560811053539, Csi creation time is 1.714674
>01:43:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 84
>01:43:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:43:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 84
>01:43:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapqgxrsq_9.yaml with timeout of 600
>01:43:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-84-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:43:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-84-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-84-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:43:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-84-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:43:43Z with timeout of 600
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:43:43Z with timeout of 600
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:43:43Z with timeout of 600
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:43:43Z with timeout of 600
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:43:43Z with timeout of 600
>01:43:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:43:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:43:43Z with timeout of 600
>01:43:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 84 are : Creation time is 1.423654 , Creation speed 5.0574086119239645, Csi creation time is 1.385722
>01:43:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 85
>01:43:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:44:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 85
>01:44:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5486r6jt.yaml with timeout of 600
>01:44:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-85-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:44:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-85-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:44:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-85-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:44:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-85-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:44:07Z with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:44:07Z with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:44:07Z with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:44:07Z with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:44:07Z with timeout of 600
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:44:07Z with timeout of 600
>01:44:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 85 are : Creation time is 1.636083 , Creation speed 4.400754729436098, Csi creation time is 1.593685
>01:44:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 86
>01:44:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:44:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 86
>01:44:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8oy3fwne.yaml with timeout of 600
>01:44:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-86-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:44:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-86-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-86-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:44:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-86-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:44:31Z with timeout of 600
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:44:31Z with timeout of 600
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:44:31Z with timeout of 600
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:44:31Z with timeout of 600
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:44:31Z with timeout of 600
>01:44:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:44:31Z with timeout of 600
>01:44:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 86 are : Creation time is 1.917503 , Creation speed 3.75488330396354, Csi creation time is 1.88029
>01:44:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 87
>01:44:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:44:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 87
>01:44:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvqfe8udt.yaml with timeout of 600
>01:44:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:44:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-87-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:44:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-87-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:45:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-87-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:45:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-87-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:45:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:45:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:44:59Z with timeout of 600
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:44:59Z with timeout of 600
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:44:59Z with timeout of 600
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:44:59Z with timeout of 600
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:44:59Z with timeout of 600
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:44:59Z with timeout of 600
>01:45:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 87 are : Creation time is 1.774747 , Creation speed 4.056916281588305, Csi creation time is 1.73493
>01:45:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 88
>01:45:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:45:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 88
>01:45:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmbdtbuzu.yaml with timeout of 600
>01:45:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-88-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:45:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-88-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-88-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:45:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-88-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:45:24Z with timeout of 600
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:45:24Z with timeout of 600
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:45:24Z with timeout of 600
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:45:24Z with timeout of 600
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:45:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:45:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:45:24Z with timeout of 600
>01:45:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:45:24Z with timeout of 600
>01:45:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 88 are : Creation time is 1.330697 , Creation speed 5.410698303220042, Csi creation time is 1.289402
>01:45:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 89
>01:45:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:45:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 89
>01:45:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_w6esbi3.yaml with timeout of 600
>01:45:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-89-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:45:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-89-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-89-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:45:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-89-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:45:48Z with timeout of 600
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:45:48Z with timeout of 600
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:45:48Z with timeout of 600
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:45:48Z with timeout of 600
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:45:48Z with timeout of 600
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:45:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:45:48Z with timeout of 600
>01:45:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 89 are : Creation time is 1.811189 , Creation speed 3.975289160877192, Csi creation time is 1.766497
>01:45:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 90
>01:45:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:46:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 90
>01:46:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbxf5u964.yaml with timeout of 600
>01:46:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-90-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:46:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-90-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-90-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:46:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-90-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:46:13Z with timeout of 600
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:46:13Z with timeout of 600
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:46:13Z with timeout of 600
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:46:13Z with timeout of 600
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:46:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:46:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:46:13Z with timeout of 600
>01:46:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:46:13Z with timeout of 600
>01:46:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 90 are : Creation time is 1.738231 , Creation speed 4.1421422123986975, Csi creation time is 1.70531
>01:46:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 91
>01:46:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:46:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 91
>01:46:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8o_dp9yd.yaml with timeout of 600
>01:46:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-91-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:46:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-91-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-91-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:46:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-91-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:46:37Z with timeout of 600
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:46:37Z with timeout of 600
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:46:37Z with timeout of 600
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:46:37Z with timeout of 600
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:46:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:46:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:46:37Z with timeout of 600
>01:46:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:46:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:46:37Z with timeout of 600
>01:46:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 91 are : Creation time is 1.787276 , Creation speed 4.028476855281445, Csi creation time is 1.748318
>01:46:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 92
>01:46:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:47:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 92
>01:47:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaptftmsmmp.yaml with timeout of 600
>01:47:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-92-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:47:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-92-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-92-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:47:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-92-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:47:01Z with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:47:01Z with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:47:01Z with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:47:01Z with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:47:01Z with timeout of 600
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:47:01Z with timeout of 600
>01:47:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 92 are : Creation time is 2.009223 , Creation speed 3.583474805932443, Csi creation time is 1.972766
>01:47:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 93
>01:47:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:47:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 93
>01:47:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapieiye0vt.yaml with timeout of 600
>01:47:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-93-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:47:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-93-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:47:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-93-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:47:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-93-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:47:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:47:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:47:26Z with timeout of 600
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:47:26Z with timeout of 600
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:47:26Z with timeout of 600
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:47:26Z with timeout of 600
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:47:26Z with timeout of 600
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:47:26Z with timeout of 600
>01:47:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 93 are : Creation time is 1.873781 , Creation speed 3.842498136121564, Csi creation time is 1.831939
>01:47:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 94
>01:47:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:47:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 94
>01:47:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapzirrxjz8.yaml with timeout of 600
>01:47:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:47:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-94-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:47:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-94-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-94-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:48:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-94-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:47:51Z with timeout of 600
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:47:51Z with timeout of 600
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:47:51Z with timeout of 600
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:47:51Z with timeout of 600
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:48:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:48:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:47:51Z with timeout of 600
>01:48:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:47:51Z with timeout of 600
>01:48:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 94 are : Creation time is 1.335654 , Creation speed 5.390617630014959, Csi creation time is 1.296068
>01:48:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 95
>01:48:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:48:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 95
>01:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapd6zdelg_.yaml with timeout of 600
>01:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-95-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:48:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-95-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-95-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:48:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-95-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:48:15Z with timeout of 600
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:48:15Z with timeout of 600
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:48:15Z with timeout of 600
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:48:15Z with timeout of 600
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:48:15Z with timeout of 600
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:48:15Z with timeout of 600
>01:48:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 95 are : Creation time is 1.208595 , Creation speed 5.957330619438273, Csi creation time is 1.1682
>01:48:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 96
>01:48:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:48:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 96
>01:48:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapiai0ikjy.yaml with timeout of 600
>01:48:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-96-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:48:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-96-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-96-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:48:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-96-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:48:38Z with timeout of 600
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:48:38Z with timeout of 600
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:48:38Z with timeout of 600
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:48:38Z with timeout of 600
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:48:38Z with timeout of 600
>01:48:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:48:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:48:38Z with timeout of 600
>01:48:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 96 are : Creation time is 1.471602 , Creation speed 4.892627218500654, Csi creation time is 1.433867
>01:48:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 97
>01:48:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:49:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 97
>01:49:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapnbf61jqe.yaml with timeout of 600
>01:49:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-97-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:49:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-97-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-97-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:49:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-97-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:49:02Z with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:49:02Z with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:49:02Z with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:49:02Z with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:49:02Z with timeout of 600
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:49:02Z with timeout of 600
>01:49:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 97 are : Creation time is 2.050828 , Creation speed 3.5107771105134122, Csi creation time is 2.00408
>01:49:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 98
>01:49:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:49:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 98
>01:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsywz2lk2.yaml with timeout of 600
>01:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-98-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:49:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-98-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-98-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:49:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-98-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:49:26Z with timeout of 600
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:49:26Z with timeout of 600
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:49:26Z with timeout of 600
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:49:26Z with timeout of 600
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:49:26Z with timeout of 600
>01:49:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:49:26Z with timeout of 600
>01:49:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 98 are : Creation time is 1.962266 , Creation speed 3.6692273116896486, Csi creation time is 1.924831
>01:49:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 99
>01:49:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:49:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 99
>01:49:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvix8mreb.yaml with timeout of 600
>01:49:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:49:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-99-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:49:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-99-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-99-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:50:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-99-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:49:51Z with timeout of 600
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:49:51Z with timeout of 600
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:49:51Z with timeout of 600
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:49:51Z with timeout of 600
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:49:51Z with timeout of 600
>01:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:49:51Z with timeout of 600
>01:50:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 99 are : Creation time is 1.387706 , Creation speed 5.1884188725854035, Csi creation time is 1.34753
>01:50:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 100
>01:50:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:50:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 100
>01:50:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwlqtylo5.yaml with timeout of 600
>01:50:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-100-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:50:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-100-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:50:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-100-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:50:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-100-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:50:14Z with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:50:14Z with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:50:14Z with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:50:14Z with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:50:14Z with timeout of 600
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:50:14Z with timeout of 600
>01:50:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 100 are : Creation time is 1.712933 , Creation speed 4.203316767205723, Csi creation time is 1.6723
>01:50:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 101
>01:50:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:50:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 101
>01:50:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapty8n78pq.yaml with timeout of 600
>01:50:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-101-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:50:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-101-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-101-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:50:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-101-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:50:39Z with timeout of 600
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:50:39Z with timeout of 600
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:50:39Z with timeout of 600
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:50:39Z with timeout of 600
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:50:39Z with timeout of 600
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:50:39Z with timeout of 600
>01:50:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 101 are : Creation time is 2.052724 , Creation speed 3.507534378708487, Csi creation time is 2.012377
>01:50:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 102
>01:50:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:51:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 102
>01:51:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap54v_1dkl.yaml with timeout of 600
>01:51:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-102-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:51:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-102-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-102-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:51:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-102-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:51:02Z with timeout of 600
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:51:02Z with timeout of 600
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:51:02Z with timeout of 600
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:51:02Z with timeout of 600
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:51:02Z with timeout of 600
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:51:02Z with timeout of 600
>01:51:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 102 are : Creation time is 2.032943 , Creation speed 3.5416634898273096, Csi creation time is 1.995658
>01:51:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 103
>01:51:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:51:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 103
>01:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3rbqlq_c.yaml with timeout of 600
>01:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-103-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:51:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-103-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-103-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:51:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-103-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:51:26Z with timeout of 600
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:51:26Z with timeout of 600
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:51:26Z with timeout of 600
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:51:26Z with timeout of 600
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:51:26Z with timeout of 600
>01:51:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:51:26Z with timeout of 600
>01:51:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 103 are : Creation time is 1.500608 , Creation speed 4.798055188297011, Csi creation time is 1.461886
>01:51:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 104
>01:51:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:51:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 104
>01:51:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphq0m9p_6.yaml with timeout of 600
>01:51:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-104-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:51:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-104-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-104-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:51:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-104-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:51:49Z with timeout of 600
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:51:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:51:49Z with timeout of 600
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:51:49Z with timeout of 600
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:51:49Z with timeout of 600
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:51:49Z with timeout of 600
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:51:49Z with timeout of 600
>01:52:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 104 are : Creation time is 1.982937 , Creation speed 3.6309776861292113, Csi creation time is 1.943526
>01:52:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 105
>01:52:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:52:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 105
>01:52:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapt_g0u_s3.yaml with timeout of 600
>01:52:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-105-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:52:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-105-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-105-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:52:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-105-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:52:13Z with timeout of 600
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:52:13Z with timeout of 600
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:52:13Z with timeout of 600
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:52:13Z with timeout of 600
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:52:13Z with timeout of 600
>01:52:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:52:13Z with timeout of 600
>01:52:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 105 are : Creation time is 1.894086 , Creation speed 3.8013057485246184, Csi creation time is 1.852228
>01:52:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 106
>01:52:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:52:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 106
>01:52:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snappb3cwbj3.yaml with timeout of 600
>01:52:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-106-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:52:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-106-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-106-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:52:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-106-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:52:36Z with timeout of 600
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:52:36Z with timeout of 600
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:52:36Z with timeout of 600
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:52:36Z with timeout of 600
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:52:36Z with timeout of 600
>01:52:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:52:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:52:36Z with timeout of 600
>01:52:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 106 are : Creation time is 1.99226 , Creation speed 3.6139861263088155, Csi creation time is 1.955654
>01:52:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 107
>01:52:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:53:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 107
>01:53:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7dj6l4r1.yaml with timeout of 600
>01:53:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-107-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:53:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-107-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-107-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:53:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-107-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:53:00Z with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:53:00Z with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:53:00Z with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:53:00Z with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:53:00Z with timeout of 600
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:53:00Z with timeout of 600
>01:53:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 107 are : Creation time is 1.616949 , Creation speed 4.452830608757605, Csi creation time is 1.571753
>01:53:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 108
>01:53:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:53:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 108
>01:53:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjt1lnp26.yaml with timeout of 600
>01:53:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-108-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:53:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-108-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-108-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:53:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-108-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:53:25Z with timeout of 600
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:53:25Z with timeout of 600
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:53:25Z with timeout of 600
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:53:25Z with timeout of 600
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:53:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:53:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:53:25Z with timeout of 600
>01:53:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:53:25Z with timeout of 600
>01:53:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 108 are : Creation time is 1.504798 , Creation speed 4.784695354459536, Csi creation time is 1.466608
>01:53:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 109
>01:53:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:53:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 109
>01:53:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapou2y5l5z.yaml with timeout of 600
>01:53:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-109-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:53:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-109-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-109-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:53:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-109-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:53:49Z with timeout of 600
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:53:49Z with timeout of 600
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:53:49Z with timeout of 600
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:53:49Z with timeout of 600
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:53:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:54:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:54:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:53:49Z with timeout of 600
>01:54:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:53:49Z with timeout of 600
>01:54:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 109 are : Creation time is 1.472357 , Creation speed 4.890118361239836, Csi creation time is 1.429031
>01:54:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 110
>01:54:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:54:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 110
>01:54:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1mxbx_3u.yaml with timeout of 600
>01:54:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-110-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:54:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-110-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-110-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:54:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-110-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:54:13Z with timeout of 600
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:54:13Z with timeout of 600
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:54:13Z with timeout of 600
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:54:13Z with timeout of 600
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:54:13Z with timeout of 600
>01:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:54:13Z with timeout of 600
>01:54:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 110 are : Creation time is 1.862774 , Creation speed 3.865203186215827, Csi creation time is 1.825208
>01:54:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 111
>01:54:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:54:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 111
>01:54:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphown0z8u.yaml with timeout of 600
>01:54:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-111-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:54:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-111-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:54:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-111-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:54:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-111-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:54:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:54:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:54:37Z with timeout of 600
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:54:37Z with timeout of 600
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:54:37Z with timeout of 600
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:54:37Z with timeout of 600
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:54:37Z with timeout of 600
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:54:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:54:37Z with timeout of 600
>01:54:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 111 are : Creation time is 1.947912 , Creation speed 3.6962655397163733, Csi creation time is 1.903018
>01:54:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 112
>01:54:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:55:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 112
>01:55:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snape1ujo7n0.yaml with timeout of 600
>01:55:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-112-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:55:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-112-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-112-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:55:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-112-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:55:01Z with timeout of 600
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:55:01Z with timeout of 600
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:55:01Z with timeout of 600
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:55:01Z with timeout of 600
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:55:01Z with timeout of 600
>01:55:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:55:01Z with timeout of 600
>01:55:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 112 are : Creation time is 1.940428 , Creation speed 3.710521596266391, Csi creation time is 1.905877
>01:55:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 113
>01:55:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:55:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 113
>01:55:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmht1q_s8.yaml with timeout of 600
>01:55:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-113-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:55:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-113-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:55:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-113-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:55:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-113-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:55:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:55:28Z with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:55:28Z with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:55:28Z with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:55:28Z with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:55:28Z with timeout of 600
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:55:28Z with timeout of 600
>01:55:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 113 are : Creation time is 1.848603 , Creation speed 3.8948330171486254, Csi creation time is 1.80809
>01:55:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 114
>01:55:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:55:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 114
>01:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwplrcak5.yaml with timeout of 600
>01:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-114-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:55:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-114-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-114-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:56:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-114-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:55:55Z with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:55:55Z with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:55:55Z with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:55:55Z with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:55:55Z with timeout of 600
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:55:55Z with timeout of 600
>01:56:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 114 are : Creation time is 1.679358 , Creation speed 4.287352666911999, Csi creation time is 1.642801
>01:56:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 115
>01:56:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:56:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 115
>01:56:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap9i_07gqn.yaml with timeout of 600
>01:56:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-115-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:56:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-115-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-115-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:56:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-115-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:56:20Z with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:56:20Z with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:56:20Z with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:56:20Z with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:56:20Z with timeout of 600
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:56:20Z with timeout of 600
>01:56:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 115 are : Creation time is 1.763541 , Creation speed 4.082694986960893, Csi creation time is 1.722136
>01:56:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 116
>01:56:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:56:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 116
>01:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap191jd9ts.yaml with timeout of 600
>01:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-116-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:56:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-116-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-116-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:56:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-116-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:56:45Z with timeout of 600
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:56:45Z with timeout of 600
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:56:45Z with timeout of 600
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:56:45Z with timeout of 600
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:56:45Z with timeout of 600
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:56:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:56:45Z with timeout of 600
>01:56:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 116 are : Creation time is 1.315352 , Creation speed 5.473819935652205, Csi creation time is 1.279936
>01:56:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 117
>01:56:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:57:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 117
>01:57:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapidz1naqe.yaml with timeout of 600
>01:57:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-117-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:57:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-117-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-117-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:57:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-117-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:57:09Z with timeout of 600
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:57:09Z with timeout of 600
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:57:09Z with timeout of 600
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:57:09Z with timeout of 600
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:57:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:57:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:57:09Z with timeout of 600
>01:57:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:57:09Z with timeout of 600
>01:57:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 117 are : Creation time is 1.482317 , Creation speed 4.8572606264382046, Csi creation time is 1.436758
>01:57:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 118
>01:57:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:57:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 118
>01:57:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapb4ksdwx5.yaml with timeout of 600
>01:57:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-118-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:57:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-118-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-118-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:57:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-118-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:57:32Z with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:57:32Z with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:57:32Z with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:57:32Z with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:57:32Z with timeout of 600
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:57:32Z with timeout of 600
>01:57:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 118 are : Creation time is 2.112949 , Creation speed 3.4075597659952988, Csi creation time is 2.077717
>01:57:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 119
>01:57:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:57:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 119
>01:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapx8_ynkxl.yaml with timeout of 600
>01:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-119-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:57:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-119-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:58:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-119-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:58:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-119-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:57:56Z with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:57:56Z with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:57:56Z with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:57:56Z with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:57:56Z with timeout of 600
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:57:56Z with timeout of 600
>01:58:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 119 are : Creation time is 1.925643 , Creation speed 3.7390108135308573, Csi creation time is 1.881937
>01:58:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 120
>01:58:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:58:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 120
>01:58:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snape1mimu0t.yaml with timeout of 600
>01:58:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-120-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:58:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-120-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:58:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-120-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:58:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-120-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:58:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:58:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:58:20Z with timeout of 600
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:58:20Z with timeout of 600
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:58:20Z with timeout of 600
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:58:20Z with timeout of 600
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:58:20Z with timeout of 600
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:58:20Z with timeout of 600
>01:58:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 120 are : Creation time is 2.084038 , Creation speed 3.4548314378144735, Csi creation time is 2.044825
>01:58:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 121
>01:58:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:58:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 121
>01:58:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmfe1j6j5.yaml with timeout of 600
>01:58:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-121-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:58:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-121-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-121-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:58:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-121-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:58:44Z with timeout of 600
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:58:44Z with timeout of 600
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:58:44Z with timeout of 600
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:58:44Z with timeout of 600
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:58:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:58:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:58:44Z with timeout of 600
>01:58:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:58:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:58:44Z with timeout of 600
>01:58:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 121 are : Creation time is 1.853205 , Creation speed 3.8851611127748953, Csi creation time is 1.811961
>01:58:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 122
>01:58:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:59:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 122
>01:59:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_1mwr8m4.yaml with timeout of 600
>01:59:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-122-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:59:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-122-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:59:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-122-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:59:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-122-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:59:07Z with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:59:07Z with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:59:07Z with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:59:07Z with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:59:07Z with timeout of 600
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:59:07Z with timeout of 600
>01:59:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 122 are : Creation time is 1.811968 , Creation speed 3.9735801073749646, Csi creation time is 1.778111
>01:59:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 123
>01:59:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:59:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 123
>01:59:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapumy8s6bp.yaml with timeout of 600
>01:59:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-123-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:59:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-123-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-123-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:59:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-123-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:59:31Z with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:59:31Z with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:59:31Z with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:59:31Z with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:59:31Z with timeout of 600
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:59:31Z with timeout of 600
>01:59:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 123 are : Creation time is 1.945491 , Creation speed 3.7008652314505697, Csi creation time is 1.908484
>01:59:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 124
>01:59:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>01:59:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 124
>01:59:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapu7e31ws4.yaml with timeout of 600
>01:59:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>01:59:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-124-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>01:59:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-124-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-124-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:00:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-124-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:59:56Z with timeout of 600
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:59:56Z with timeout of 600
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T09:59:56Z with timeout of 600
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T09:59:56Z with timeout of 600
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:00:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:00:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T09:59:56Z with timeout of 600
>02:00:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T09:59:56Z with timeout of 600
>02:00:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 124 are : Creation time is 1.643889 , Creation speed 4.3798577641191105, Csi creation time is 1.605092
>02:00:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 125
>02:00:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:00:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 125
>02:00:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapib3qv0z4.yaml with timeout of 600
>02:00:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-125-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:00:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-125-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-125-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:00:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-125-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:00:20Z with timeout of 600
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:00:20Z with timeout of 600
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:00:20Z with timeout of 600
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:00:20Z with timeout of 600
>02:00:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:00:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:00:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:00:20Z with timeout of 600
>02:00:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:00:20Z with timeout of 600
>02:00:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 125 are : Creation time is 1.537933 , Creation speed 4.68160836655433, Csi creation time is 1.493332
>02:00:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 126
>02:00:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:00:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 126
>02:00:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap120iwanz.yaml with timeout of 600
>02:00:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-126-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:00:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-126-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-126-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:00:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-126-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:00:44Z with timeout of 600
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:00:44Z with timeout of 600
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:00:44Z with timeout of 600
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:00:44Z with timeout of 600
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:00:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:00:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:00:44Z with timeout of 600
>02:00:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:00:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:00:44Z with timeout of 600
>02:00:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 126 are : Creation time is 2.118261 , Creation speed 3.3990145690261966, Csi creation time is 2.077658
>02:00:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 127
>02:00:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:01:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 127
>02:01:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapu4cg6lxu.yaml with timeout of 600
>02:01:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-127-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:01:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-127-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-127-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:01:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-127-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:01:08Z with timeout of 600
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:01:08Z with timeout of 600
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:01:08Z with timeout of 600
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:01:08Z with timeout of 600
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:01:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:01:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:01:08Z with timeout of 600
>02:01:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:01:08Z with timeout of 600
>02:01:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 127 are : Creation time is 1.18572 , Creation speed 6.072259892723408, Csi creation time is 1.145815
>02:01:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 128
>02:01:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:01:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 128
>02:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4sjbgjjl.yaml with timeout of 600
>02:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-128-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:01:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-128-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-128-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:01:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-128-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:01:32Z with timeout of 600
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:01:32Z with timeout of 600
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:01:32Z with timeout of 600
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:01:32Z with timeout of 600
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:01:32Z with timeout of 600
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:01:32Z with timeout of 600
>02:01:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 128 are : Creation time is 1.496774 , Creation speed 4.810345449613636, Csi creation time is 1.457464
>02:01:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 129
>02:01:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:01:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 129
>02:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapurt8y6zd.yaml with timeout of 600
>02:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-129-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:01:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-129-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-129-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:02:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-129-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:01:56Z with timeout of 600
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:01:56Z with timeout of 600
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:01:56Z with timeout of 600
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:01:56Z with timeout of 600
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:01:56Z with timeout of 600
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:01:56Z with timeout of 600
>02:02:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 129 are : Creation time is 2.134195 , Creation speed 3.373637366782323, Csi creation time is 2.09548
>02:02:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 130
>02:02:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:02:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 130
>02:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphags4iju.yaml with timeout of 600
>02:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-130-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:02:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-130-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-130-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:02:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-130-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:02:20Z with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:02:20Z with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:02:20Z with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:02:20Z with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:02:20Z with timeout of 600
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:02:20Z with timeout of 600
>02:02:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 130 are : Creation time is 1.929595 , Creation speed 3.731352952303463, Csi creation time is 1.892549
>02:02:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 131
>02:02:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:02:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 131
>02:02:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap2z0agkk5.yaml with timeout of 600
>02:02:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-131-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:02:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-131-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:02:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-131-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:02:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-131-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:02:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:02:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:02:44Z with timeout of 600
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:02:44Z with timeout of 600
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:02:44Z with timeout of 600
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:02:44Z with timeout of 600
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:02:44Z with timeout of 600
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:02:44Z with timeout of 600
>02:02:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 131 are : Creation time is 2.098308 , Creation speed 3.4313361050903874, Csi creation time is 2.051598
>02:02:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 132
>02:02:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:03:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 132
>02:03:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapzwarogza.yaml with timeout of 600
>02:03:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-132-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:03:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-132-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:03:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-132-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:03:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-132-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:03:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:03:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:03:08Z with timeout of 600
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:03:08Z with timeout of 600
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:03:08Z with timeout of 600
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:03:08Z with timeout of 600
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:03:08Z with timeout of 600
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:03:08Z with timeout of 600
>02:03:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 132 are : Creation time is 1.18079 , Creation speed 6.097612615282989, Csi creation time is 1.143431
>02:03:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 133
>02:03:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:03:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 133
>02:03:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snappz3exoa2.yaml with timeout of 600
>02:03:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-133-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:03:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-133-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-133-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:03:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-133-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:03:32Z with timeout of 600
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:03:32Z with timeout of 600
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:03:32Z with timeout of 600
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:03:32Z with timeout of 600
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:03:32Z with timeout of 600
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:03:32Z with timeout of 600
>02:03:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 133 are : Creation time is 1.497019 , Creation speed 4.809558195320166, Csi creation time is 1.458705
>02:03:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 134
>02:03:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:03:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 134
>02:03:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmkc6i87x.yaml with timeout of 600
>02:03:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:03:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-134-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:03:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-134-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-134-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:04:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-134-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:03:56Z with timeout of 600
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:03:56Z with timeout of 600
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:03:56Z with timeout of 600
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:03:56Z with timeout of 600
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:03:56Z with timeout of 600
>02:04:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:03:56Z with timeout of 600
>02:04:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 134 are : Creation time is 2.058775 , Creation speed 3.49722529173902, Csi creation time is 2.020814
>02:04:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 135
>02:04:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:04:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 135
>02:04:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapo8ebukr7.yaml with timeout of 600
>02:04:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-135-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:04:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-135-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-135-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:04:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-135-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:04:20Z with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:04:20Z with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:04:20Z with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:04:20Z with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:04:20Z with timeout of 600
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:04:20Z with timeout of 600
>02:04:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 135 are : Creation time is 1.206004 , Creation speed 5.970129452306957, Csi creation time is 1.162951
>02:04:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 136
>02:04:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:04:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 136
>02:04:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap9so6qxv8.yaml with timeout of 600
>02:04:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-136-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:04:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-136-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-136-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:04:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-136-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:04:47Z with timeout of 600
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:04:47Z with timeout of 600
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:04:47Z with timeout of 600
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:04:47Z with timeout of 600
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:04:47Z with timeout of 600
>02:04:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:04:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:04:47Z with timeout of 600
>02:04:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 136 are : Creation time is 2.137127 , Creation speed 3.3690089545450506, Csi creation time is 2.054652
>02:04:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 137
>02:04:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:05:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 137
>02:05:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjuwzfm7p.yaml with timeout of 600
>02:05:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-137-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:05:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-137-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-137-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:05:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-137-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:05:14Z with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:05:14Z with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:05:14Z with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:05:14Z with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:05:14Z with timeout of 600
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:05:14Z with timeout of 600
>02:05:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 137 are : Creation time is 1.423781 , Creation speed 5.056957495569895, Csi creation time is 1.38364
>02:05:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 138
>02:05:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:05:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 138
>02:05:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapch7f7w49.yaml with timeout of 600
>02:05:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-138-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:05:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-138-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-138-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:05:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-138-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:05:39Z with timeout of 600
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:05:39Z with timeout of 600
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:05:39Z with timeout of 600
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:05:39Z with timeout of 600
>02:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:05:39Z with timeout of 600
>02:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:05:39Z with timeout of 600
>02:05:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 138 are : Creation time is 1.977574 , Creation speed 3.640824565856954, Csi creation time is 1.937984
>02:05:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 139
>02:05:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:06:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 139
>02:06:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapg_ldaodu.yaml with timeout of 600
>02:06:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-139-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:06:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-139-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-139-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:06:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-139-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:06:03Z with timeout of 600
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:06:03Z with timeout of 600
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:06:03Z with timeout of 600
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:06:03Z with timeout of 600
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:06:03Z with timeout of 600
>02:06:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:06:03Z with timeout of 600
>02:06:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 139 are : Creation time is 1.245715 , Creation speed 5.7798131996483955, Csi creation time is 1.204285
>02:06:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 140
>02:06:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:06:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 140
>02:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1zuo4w6c.yaml with timeout of 600
>02:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-140-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:06:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-140-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:06:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-140-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:06:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-140-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:06:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:06:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:06:28Z with timeout of 600
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:06:28Z with timeout of 600
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:06:28Z with timeout of 600
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:06:28Z with timeout of 600
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:06:28Z with timeout of 600
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:06:28Z with timeout of 600
>02:06:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 140 are : Creation time is 1.682919 , Creation speed 4.278280772871422, Csi creation time is 1.643502
>02:06:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 141
>02:06:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:06:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 141
>02:06:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapk1gtysgr.yaml with timeout of 600
>02:06:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:06:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-141-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:06:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-141-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-141-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:07:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-141-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:06:52Z with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:06:52Z with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:06:52Z with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:06:52Z with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:06:52Z with timeout of 600
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:06:52Z with timeout of 600
>02:07:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 141 are : Creation time is 1.528388 , Creation speed 4.710845675312813, Csi creation time is 1.48741
>02:07:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 142
>02:07:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:07:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 142
>02:07:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapcd95jgma.yaml with timeout of 600
>02:07:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-142-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:07:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-142-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:07:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-142-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:07:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-142-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:07:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:07:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:07:16Z with timeout of 600
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:07:16Z with timeout of 600
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:07:16Z with timeout of 600
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:07:16Z with timeout of 600
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:07:16Z with timeout of 600
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:07:16Z with timeout of 600
>02:07:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 142 are : Creation time is 1.772381 , Creation speed 4.062331970383343, Csi creation time is 1.730769
>02:07:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 143
>02:07:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:07:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 143
>02:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapcd68qs5v.yaml with timeout of 600
>02:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-143-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:07:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-143-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-143-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:07:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-143-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:07:43Z with timeout of 600
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:07:43Z with timeout of 600
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:07:43Z with timeout of 600
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:07:43Z with timeout of 600
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:07:43Z with timeout of 600
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:07:43Z with timeout of 600
>02:07:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 143 are : Creation time is 2.025017 , Creation speed 3.555525706697771, Csi creation time is 1.987605
>02:07:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 144
>02:07:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:08:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 144
>02:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapx6p43h8i.yaml with timeout of 600
>02:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-144-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:08:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-144-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-144-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:08:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-144-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:08:07Z with timeout of 600
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:08:07Z with timeout of 600
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:08:07Z with timeout of 600
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:08:07Z with timeout of 600
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:08:07Z with timeout of 600
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:08:07Z with timeout of 600
>02:08:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 144 are : Creation time is 1.767598 , Creation speed 4.073324364476538, Csi creation time is 1.731884
>02:08:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 145
>02:08:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:08:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 145
>02:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_oj24t5i.yaml with timeout of 600
>02:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-145-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:08:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-145-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-145-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:08:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-145-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:08:31Z with timeout of 600
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:08:31Z with timeout of 600
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:08:31Z with timeout of 600
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:08:31Z with timeout of 600
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:08:31Z with timeout of 600
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:08:31Z with timeout of 600
>02:08:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 145 are : Creation time is 1.514785 , Creation speed 4.753149786933459, Csi creation time is 1.474219
>02:08:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 146
>02:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:08:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 146
>02:08:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfowbwtxw.yaml with timeout of 600
>02:08:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:08:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-146-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:08:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-146-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-146-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:09:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-146-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:08:55Z with timeout of 600
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:08:55Z with timeout of 600
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:08:55Z with timeout of 600
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:08:55Z with timeout of 600
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:08:55Z with timeout of 600
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:08:55Z with timeout of 600
>02:09:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 146 are : Creation time is 1.907281 , Creation speed 3.775007458261263, Csi creation time is 1.868586
>02:09:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 147
>02:09:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:09:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 147
>02:09:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4jqis80a.yaml with timeout of 600
>02:09:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-147-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:09:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-147-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-147-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:09:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-147-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:09:19Z with timeout of 600
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:09:19Z with timeout of 600
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:09:19Z with timeout of 600
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:09:19Z with timeout of 600
>02:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:09:19Z with timeout of 600
>02:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:09:19Z with timeout of 600
>02:09:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 147 are : Creation time is 2.002197 , Creation speed 3.596049739361312, Csi creation time is 1.962637
>02:09:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 148
>02:09:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:09:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 148
>02:09:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1ufms45g.yaml with timeout of 600
>02:09:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-148-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:09:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-148-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-148-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:09:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-148-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:09:42Z with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:09:42Z with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:09:42Z with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:09:42Z with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:09:42Z with timeout of 600
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:09:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:09:42Z with timeout of 600
>02:09:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 148 are : Creation time is 1.418987 , Creation speed 5.074042256905807, Csi creation time is 1.374385
>02:09:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 149
>02:09:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:10:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 149
>02:10:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapeb60n1i4.yaml with timeout of 600
>02:10:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-149-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:10:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-149-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:10:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-149-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:10:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-149-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:10:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:10:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:10:06Z with timeout of 600
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:10:06Z with timeout of 600
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:10:06Z with timeout of 600
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:10:06Z with timeout of 600
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:10:06Z with timeout of 600
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:10:06Z with timeout of 600
>02:10:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 149 are : Creation time is 1.713199 , Creation speed 4.202664138842015, Csi creation time is 1.673595
>02:10:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 150
>02:10:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:10:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 150
>02:10:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap68q31nzl.yaml with timeout of 600
>02:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-150-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:10:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-150-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-150-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:10:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-150-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:10:30Z with timeout of 600
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:10:30Z with timeout of 600
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:10:30Z with timeout of 600
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:10:30Z with timeout of 600
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:10:30Z with timeout of 600
>02:10:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:10:30Z with timeout of 600
>02:10:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 150 are : Creation time is 1.438461 , Creation speed 5.005349467243116, Csi creation time is 1.40433
>02:10:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 151
>02:10:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:10:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 151
>02:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdojf4e3b.yaml with timeout of 600
>02:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:10:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-151-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:10:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-151-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-151-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:11:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-151-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:10:55Z with timeout of 600
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:10:55Z with timeout of 600
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:10:55Z with timeout of 600
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:10:55Z with timeout of 600
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:10:55Z with timeout of 600
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:10:55Z with timeout of 600
>02:11:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 151 are : Creation time is 2.034624 , Creation speed 3.5387373785033502, Csi creation time is 1.991313
>02:11:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 152
>02:11:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:11:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 152
>02:11:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapphz1ipc5.yaml with timeout of 600
>02:11:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-152-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:11:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-152-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-152-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:11:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-152-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:11:19Z with timeout of 600
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:11:19Z with timeout of 600
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:11:19Z with timeout of 600
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:11:19Z with timeout of 600
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:11:19Z with timeout of 600
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:11:19Z with timeout of 600
>02:11:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 152 are : Creation time is 1.204899 , Creation speed 5.9756045942439995, Csi creation time is 1.163876
>02:11:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 153
>02:11:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:11:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 153
>02:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_lfs21u3.yaml with timeout of 600
>02:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-153-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:11:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-153-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-153-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:11:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-153-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:11:42Z with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:11:42Z with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:11:42Z with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:11:42Z with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:11:42Z with timeout of 600
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:11:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:11:42Z with timeout of 600
>02:11:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 153 are : Creation time is 1.611559 , Creation speed 4.467723490111129, Csi creation time is 1.572089
>02:11:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 154
>02:11:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:12:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 154
>02:12:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaps4608695.yaml with timeout of 600
>02:12:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-154-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:12:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-154-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-154-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:12:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-154-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:12:06Z with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:12:06Z with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:12:06Z with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:12:06Z with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:12:06Z with timeout of 600
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:12:06Z with timeout of 600
>02:12:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 154 are : Creation time is 1.710553 , Creation speed 4.209165106255112, Csi creation time is 1.66941
>02:12:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 155
>02:12:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:12:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 155
>02:12:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1nmo09_r.yaml with timeout of 600
>02:12:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-155-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:12:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-155-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-155-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:12:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-155-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:12:30Z with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:12:30Z with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:12:30Z with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:12:30Z with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:12:30Z with timeout of 600
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:12:30Z with timeout of 600
>02:12:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 155 are : Creation time is 1.74853 , Creation speed 4.117744619766318, Csi creation time is 1.707737
>02:12:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 156
>02:12:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:12:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 156
>02:12:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapb9pald1e.yaml with timeout of 600
>02:12:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:12:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-156-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:12:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-156-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-156-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:13:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-156-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:12:55Z with timeout of 600
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:12:55Z with timeout of 600
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:12:55Z with timeout of 600
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:12:55Z with timeout of 600
>02:13:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:13:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:13:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:12:55Z with timeout of 600
>02:13:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:12:55Z with timeout of 600
>02:13:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 156 are : Creation time is 1.497922 , Creation speed 4.806658824691807, Csi creation time is 1.462283
>02:13:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 157
>02:13:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:13:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 157
>02:13:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap297wio88.yaml with timeout of 600
>02:13:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-157-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:13:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-157-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-157-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:13:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-157-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:13:21Z with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:13:21Z with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:13:21Z with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:13:21Z with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:13:21Z with timeout of 600
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:13:21Z with timeout of 600
>02:13:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 157 are : Creation time is 1.469399 , Creation speed 4.899962501675856, Csi creation time is 1.427866
>02:13:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 158
>02:13:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:13:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 158
>02:13:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapadsxavux.yaml with timeout of 600
>02:13:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-158-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:13:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-158-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-158-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:13:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-158-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:13:45Z with timeout of 600
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:13:45Z with timeout of 600
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:13:45Z with timeout of 600
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:13:45Z with timeout of 600
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:13:45Z with timeout of 600
>02:13:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:13:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:13:45Z with timeout of 600
>02:13:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 158 are : Creation time is 1.272117 , Creation speed 5.659856758458538, Csi creation time is 1.235079
>02:13:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 159
>02:13:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:14:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 159
>02:14:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphkulhys7.yaml with timeout of 600
>02:14:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-159-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:14:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-159-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-159-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:14:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-159-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:14:10Z with timeout of 600
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:14:10Z with timeout of 600
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:14:10Z with timeout of 600
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:14:10Z with timeout of 600
>02:14:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:14:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:14:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:14:10Z with timeout of 600
>02:14:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:14:10Z with timeout of 600
>02:14:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 159 are : Creation time is 2.080024 , Creation speed 3.4614985211709097, Csi creation time is 2.03959
>02:14:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 160
>02:14:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:14:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 160
>02:14:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapg2hahxru.yaml with timeout of 600
>02:14:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-160-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:14:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-160-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:14:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-160-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:14:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-160-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:14:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:14:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:14:34Z with timeout of 600
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:14:34Z with timeout of 600
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:14:34Z with timeout of 600
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:14:34Z with timeout of 600
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:14:34Z with timeout of 600
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:14:34Z with timeout of 600
>02:14:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 160 are : Creation time is 1.764477 , Creation speed 4.080529244643031, Csi creation time is 1.723891
>02:14:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 161
>02:14:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:14:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 161
>02:14:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapqnve4wfj.yaml with timeout of 600
>02:14:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:14:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-161-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:14:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-161-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:15:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-161-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:15:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-161-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:15:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:14:58Z with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:14:58Z with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:14:58Z with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:14:58Z with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:14:58Z with timeout of 600
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:14:58Z with timeout of 600
>02:15:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 161 are : Creation time is 1.728822 , Creation speed 4.164685548888202, Csi creation time is 1.688171
>02:15:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 162
>02:15:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:15:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 162
>02:15:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0227qu47.yaml with timeout of 600
>02:15:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-162-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:15:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-162-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-162-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:15:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-162-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:15:23Z with timeout of 600
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:15:23Z with timeout of 600
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:15:23Z with timeout of 600
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:15:23Z with timeout of 600
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:15:23Z with timeout of 600
>02:15:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:15:23Z with timeout of 600
>02:15:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 162 are : Creation time is 1.432238 , Creation speed 5.027097451680517, Csi creation time is 1.393456
>02:15:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 163
>02:15:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:15:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 163
>02:15:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsy7f1kxu.yaml with timeout of 600
>02:15:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:15:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-163-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:15:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-163-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-163-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:16:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-163-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:15:50Z with timeout of 600
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:15:50Z with timeout of 600
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:15:50Z with timeout of 600
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:15:50Z with timeout of 600
>02:16:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:16:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:16:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:15:50Z with timeout of 600
>02:16:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:15:50Z with timeout of 600
>02:16:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 163 are : Creation time is 1.222275 , Creation speed 5.890654721727926, Csi creation time is 1.184522
>02:16:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 164
>02:16:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:16:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 164
>02:16:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapv89bx1dp.yaml with timeout of 600
>02:16:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-164-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:16:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-164-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:16:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-164-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:16:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-164-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:16:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:16:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:16:14Z with timeout of 600
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:16:14Z with timeout of 600
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:16:14Z with timeout of 600
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:16:14Z with timeout of 600
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:16:14Z with timeout of 600
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:16:14Z with timeout of 600
>02:16:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 164 are : Creation time is 1.944073 , Creation speed 3.7035646295175133, Csi creation time is 1.907225
>02:16:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 165
>02:16:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:16:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 165
>02:16:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapv5zbepua.yaml with timeout of 600
>02:16:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-165-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:16:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-165-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:16:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-165-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:16:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-165-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:16:38Z with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:16:38Z with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:16:38Z with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:16:38Z with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:16:38Z with timeout of 600
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:16:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:16:38Z with timeout of 600
>02:16:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 165 are : Creation time is 1.772342 , Creation speed 4.062421361114277, Csi creation time is 1.73441
>02:16:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 166
>02:16:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:17:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 166
>02:17:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapl9iuaxdx.yaml with timeout of 600
>02:17:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-166-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:17:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-166-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-166-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:17:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-166-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:17:03Z with timeout of 600
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:17:03Z with timeout of 600
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:17:03Z with timeout of 600
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:17:03Z with timeout of 600
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:17:03Z with timeout of 600
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:17:03Z with timeout of 600
>02:17:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 166 are : Creation time is 1.31868 , Creation speed 5.46000546000546, Csi creation time is 1.276325
>02:17:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 167
>02:17:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:17:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 167
>02:17:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapymkv3w0d.yaml with timeout of 600
>02:17:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-167-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:17:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-167-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-167-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:17:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-167-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:17:27Z with timeout of 600
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:17:27Z with timeout of 600
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:17:27Z with timeout of 600
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:17:27Z with timeout of 600
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:17:27Z with timeout of 600
>02:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:17:27Z with timeout of 600
>02:17:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 167 are : Creation time is 1.732231 , Creation speed 4.15648952131673, Csi creation time is 1.692741
>02:17:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 168
>02:17:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:17:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 168
>02:17:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapyele6k4h.yaml with timeout of 600
>02:17:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:17:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-168-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:17:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-168-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:18:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-168-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:18:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-168-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:17:52Z with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:17:52Z with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:17:52Z with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:17:52Z with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:17:52Z with timeout of 600
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:17:52Z with timeout of 600
>02:18:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 168 are : Creation time is 1.971945 , Creation speed 3.6512174528194246, Csi creation time is 1.934447
>02:18:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 169
>02:18:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:18:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 169
>02:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapb4asla5z.yaml with timeout of 600
>02:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-169-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:18:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-169-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-169-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:18:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-169-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:18:17Z with timeout of 600
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:18:17Z with timeout of 600
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:18:17Z with timeout of 600
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:18:17Z with timeout of 600
>02:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:18:17Z with timeout of 600
>02:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:18:17Z with timeout of 600
>02:18:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 169 are : Creation time is 1.924878 , Creation speed 3.7404968003166954, Csi creation time is 1.883787
>02:18:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 170
>02:18:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:18:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 170
>02:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_v0yrnku.yaml with timeout of 600
>02:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-170-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:18:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-170-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-170-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:18:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-170-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:18:41Z with timeout of 600
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:18:41Z with timeout of 600
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:18:41Z with timeout of 600
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:18:41Z with timeout of 600
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:18:41Z with timeout of 600
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:18:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:18:41Z with timeout of 600
>02:18:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 170 are : Creation time is 1.757469 , Creation speed 4.096800569455279, Csi creation time is 1.717294
>02:18:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 171
>02:18:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:19:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 171
>02:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbtsdbw_g.yaml with timeout of 600
>02:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-171-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:19:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-171-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-171-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:19:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-171-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:19:06Z with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:19:06Z with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:19:06Z with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:19:06Z with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:19:06Z with timeout of 600
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:19:06Z with timeout of 600
>02:19:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 171 are : Creation time is 1.502133 , Creation speed 4.793184092220862, Csi creation time is 1.453303
>02:19:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 172
>02:19:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:19:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 172
>02:19:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapcxxuo11y.yaml with timeout of 600
>02:19:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-172-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:19:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-172-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-172-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:19:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-172-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:19:34Z with timeout of 600
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:19:34Z with timeout of 600
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:19:34Z with timeout of 600
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:19:34Z with timeout of 600
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:19:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:19:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:19:34Z with timeout of 600
>02:19:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:19:34Z with timeout of 600
>02:19:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 172 are : Creation time is 1.325169 , Creation speed 5.433269266033238, Csi creation time is 1.287734
>02:19:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 173
>02:19:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:19:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 173
>02:19:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapupepiid2.yaml with timeout of 600
>02:19:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:19:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-173-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:19:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-173-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-173-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:20:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-173-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:19:59Z with timeout of 600
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:19:59Z with timeout of 600
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:19:59Z with timeout of 600
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:19:59Z with timeout of 600
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:19:59Z with timeout of 600
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:19:59Z with timeout of 600
>02:20:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 173 are : Creation time is 1.89693 , Creation speed 3.795606585377426, Csi creation time is 1.85804
>02:20:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 174
>02:20:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:20:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 174
>02:20:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapj_pq4it9.yaml with timeout of 600
>02:20:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-174-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:20:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-174-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-174-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:20:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-174-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:20:24Z with timeout of 600
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:20:24Z with timeout of 600
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:20:24Z with timeout of 600
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:20:24Z with timeout of 600
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:20:24Z with timeout of 600
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:20:24Z with timeout of 600
>02:20:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 174 are : Creation time is 2.106071 , Creation speed 3.4186881638843136, Csi creation time is 2.065245
>02:20:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 175
>02:20:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:20:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 175
>02:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapo7c6kus6.yaml with timeout of 600
>02:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-175-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:20:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-175-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:20:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-175-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:20:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-175-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:20:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:20:48Z with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:20:48Z with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:20:48Z with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:20:48Z with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:20:48Z with timeout of 600
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:20:48Z with timeout of 600
>02:20:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 175 are : Creation time is 1.807109 , Creation speed 3.984264369221779, Csi creation time is 1.764262
>02:20:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 176
>02:20:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:21:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 176
>02:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1oom99p3.yaml with timeout of 600
>02:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-176-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:21:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-176-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-176-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:21:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-176-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:21:13Z with timeout of 600
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:21:13Z with timeout of 600
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:21:13Z with timeout of 600
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:21:13Z with timeout of 600
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:21:13Z with timeout of 600
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:21:13Z with timeout of 600
>02:21:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 176 are : Creation time is 2.067417 , Creation speed 3.4826065568774953, Csi creation time is 2.0331
>02:21:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 177
>02:21:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:21:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 177
>02:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaptewvkl3k.yaml with timeout of 600
>02:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-177-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:21:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-177-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:21:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-177-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:21:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-177-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:21:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:21:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:21:37Z with timeout of 600
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:21:37Z with timeout of 600
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:21:37Z with timeout of 600
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:21:37Z with timeout of 600
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:21:37Z with timeout of 600
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:21:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:21:37Z with timeout of 600
>02:21:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 177 are : Creation time is 1.653555 , Creation speed 4.354254923483041, Csi creation time is 1.615232
>02:21:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 178
>02:21:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:22:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 178
>02:22:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap20d5b9hl.yaml with timeout of 600
>02:22:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-178-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:22:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-178-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-178-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:22:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-178-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:22:01Z with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:22:01Z with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:22:01Z with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:22:01Z with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:22:01Z with timeout of 600
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:22:01Z with timeout of 600
>02:22:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 178 are : Creation time is 1.393554 , Creation speed 5.1666458565653, Csi creation time is 1.356789
>02:22:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 179
>02:22:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:22:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 179
>02:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjt3009e1.yaml with timeout of 600
>02:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-179-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:22:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-179-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-179-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:22:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-179-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:22:26Z with timeout of 600
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:22:26Z with timeout of 600
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:22:26Z with timeout of 600
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:22:26Z with timeout of 600
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:22:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:22:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:22:26Z with timeout of 600
>02:22:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:22:26Z with timeout of 600
>02:22:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 179 are : Creation time is 1.189648 , Creation speed 6.052210401732277, Csi creation time is 1.147498
>02:22:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 180
>02:22:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:22:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 180
>02:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapli1mpe5p.yaml with timeout of 600
>02:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-180-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:22:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-180-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-180-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:23:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-180-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:22:50Z with timeout of 600
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:22:50Z with timeout of 600
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:22:50Z with timeout of 600
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:22:50Z with timeout of 600
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:22:50Z with timeout of 600
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:22:50Z with timeout of 600
>02:23:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 180 are : Creation time is 2.03065 , Creation speed 3.5456627188338707, Csi creation time is 1.985808
>02:23:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 181
>02:23:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:23:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 181
>02:23:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapquk935n1.yaml with timeout of 600
>02:23:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-181-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:23:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-181-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:23:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-181-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:23:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-181-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:23:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:23:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:23:14Z with timeout of 600
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:23:14Z with timeout of 600
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:23:14Z with timeout of 600
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:23:14Z with timeout of 600
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:23:14Z with timeout of 600
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:23:14Z with timeout of 600
>02:23:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 181 are : Creation time is 1.785067 , Creation speed 4.033462049323639, Csi creation time is 1.741097
>02:23:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 182
>02:23:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:23:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 182
>02:23:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprslp5_ue.yaml with timeout of 600
>02:23:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-182-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:23:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-182-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-182-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:23:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-182-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:23:38Z with timeout of 600
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:23:38Z with timeout of 600
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:23:38Z with timeout of 600
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:23:38Z with timeout of 600
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:23:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:23:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:23:38Z with timeout of 600
>02:23:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:23:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:23:38Z with timeout of 600
>02:23:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 182 are : Creation time is 1.484069 , Creation speed 4.851526445198976, Csi creation time is 1.450761
>02:23:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 183
>02:23:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:24:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 183
>02:24:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaps76ezkuc.yaml with timeout of 600
>02:24:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-183-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:24:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-183-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-183-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:24:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-183-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:24:02Z with timeout of 600
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:24:02Z with timeout of 600
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:24:02Z with timeout of 600
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:24:02Z with timeout of 600
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:24:02Z with timeout of 600
>02:24:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:24:02Z with timeout of 600
>02:24:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 183 are : Creation time is 1.55254 , Creation speed 4.637561673129195, Csi creation time is 1.514157
>02:24:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 184
>02:24:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:24:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 184
>02:24:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8yq15dni.yaml with timeout of 600
>02:24:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-184-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:24:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-184-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:24:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-184-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:24:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-184-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:24:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:24:27Z with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:24:27Z with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:24:27Z with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:24:27Z with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:24:27Z with timeout of 600
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:24:27Z with timeout of 600
>02:24:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 184 are : Creation time is 1.847407 , Creation speed 3.8973545082377625, Csi creation time is 1.810778
>02:24:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 185
>02:24:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:24:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 185
>02:24:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapkombaklt.yaml with timeout of 600
>02:24:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:24:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-185-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:24:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-185-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:25:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-185-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:25:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-185-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:25:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:25:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:24:52Z with timeout of 600
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:24:52Z with timeout of 600
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:24:52Z with timeout of 600
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:24:52Z with timeout of 600
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:24:52Z with timeout of 600
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:24:52Z with timeout of 600
>02:25:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 185 are : Creation time is 2.020346 , Creation speed 3.5637460118217374, Csi creation time is 1.979634
>02:25:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 186
>02:25:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:25:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 186
>02:25:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapzzojsunw.yaml with timeout of 600
>02:25:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-186-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:25:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-186-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-186-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:25:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-186-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:25:16Z with timeout of 600
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:25:16Z with timeout of 600
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:25:16Z with timeout of 600
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:25:16Z with timeout of 600
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:25:16Z with timeout of 600
>02:25:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:25:16Z with timeout of 600
>02:25:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 186 are : Creation time is 1.713229 , Creation speed 4.202590546856259, Csi creation time is 1.670628
>02:25:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 187
>02:25:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:25:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 187
>02:25:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snappnxgef1m.yaml with timeout of 600
>02:25:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-187-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:25:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-187-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-187-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:25:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-187-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:25:42Z with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:25:42Z with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:25:42Z with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:25:42Z with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:25:42Z with timeout of 600
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:25:42Z with timeout of 600
>02:25:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 187 are : Creation time is 1.428405 , Creation speed 5.04058722841211, Csi creation time is 1.386007
>02:25:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 188
>02:25:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:26:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 188
>02:26:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphpn2geh6.yaml with timeout of 600
>02:26:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-188-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:26:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-188-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:26:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-188-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:26:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-188-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:26:08Z with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:26:08Z with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:26:08Z with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:26:08Z with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:26:08Z with timeout of 600
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:26:08Z with timeout of 600
>02:26:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 188 are : Creation time is 1.50415 , Creation speed 4.7867566399627695, Csi creation time is 1.470267
>02:26:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 189
>02:26:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:26:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 189
>02:26:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_s6nweh3.yaml with timeout of 600
>02:26:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-189-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:26:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-189-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:26:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-189-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:26:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-189-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:26:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:26:33Z with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:26:33Z with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:26:33Z with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:26:33Z with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:26:33Z with timeout of 600
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:26:33Z with timeout of 600
>02:26:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 189 are : Creation time is 1.713503 , Creation speed 4.201918525966981, Csi creation time is 1.673312
>02:26:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 190
>02:26:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:26:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 190
>02:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap9rsk1_op.yaml with timeout of 600
>02:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:26:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-190-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:26:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-190-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-190-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:27:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-190-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:26:58Z with timeout of 600
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:26:58Z with timeout of 600
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:26:58Z with timeout of 600
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:26:58Z with timeout of 600
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:26:58Z with timeout of 600
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:26:58Z with timeout of 600
>02:27:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 190 are : Creation time is 2.114305 , Creation speed 3.40537434286917, Csi creation time is 2.074352
>02:27:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 191
>02:27:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:27:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 191
>02:27:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapoa85njg9.yaml with timeout of 600
>02:27:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-191-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:27:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-191-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:27:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-191-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:27:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-191-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:27:25Z with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:27:25Z with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:27:25Z with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:27:25Z with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:27:25Z with timeout of 600
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:27:25Z with timeout of 600
>02:27:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 191 are : Creation time is 1.65614 , Creation speed 4.347458548190371, Csi creation time is 1.617124
>02:27:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 192
>02:27:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:27:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 192
>02:27:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapowbgyp6p.yaml with timeout of 600
>02:27:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:27:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-192-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:27:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-192-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:28:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-192-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:28:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-192-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:27:51Z with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:27:51Z with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:27:51Z with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:27:51Z with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:27:51Z with timeout of 600
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:27:51Z with timeout of 600
>02:28:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 192 are : Creation time is 1.789224 , Creation speed 4.024090890799587, Csi creation time is 1.750763
>02:28:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 193
>02:28:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:28:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 193
>02:28:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapj8av_htg.yaml with timeout of 600
>02:28:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-193-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:28:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-193-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-193-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:28:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-193-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:28:16Z with timeout of 600
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:28:16Z with timeout of 600
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:28:16Z with timeout of 600
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:28:16Z with timeout of 600
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:28:16Z with timeout of 600
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:28:16Z with timeout of 600
>02:28:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 193 are : Creation time is 2.13474 , Creation speed 3.372776075775036, Csi creation time is 2.095073
>02:28:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 194
>02:28:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:28:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 194
>02:28:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwsminigi.yaml with timeout of 600
>02:28:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-194-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:28:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-194-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-194-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:28:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-194-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:28:41Z with timeout of 600
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:28:41Z with timeout of 600
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:28:41Z with timeout of 600
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:28:41Z with timeout of 600
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:28:41Z with timeout of 600
>02:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:28:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:28:41Z with timeout of 600
>02:28:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 194 are : Creation time is 1.495962 , Creation speed 4.81295647884104, Csi creation time is 1.458857
>02:28:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 195
>02:28:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:29:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 195
>02:29:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprdjqnvuj.yaml with timeout of 600
>02:29:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-195-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:29:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-195-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-195-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:29:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-195-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:29:05Z with timeout of 600
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:29:05Z with timeout of 600
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:29:05Z with timeout of 600
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:29:05Z with timeout of 600
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:29:05Z with timeout of 600
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:29:05Z with timeout of 600
>02:29:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 195 are : Creation time is 1.181103 , Creation speed 6.095996708161778, Csi creation time is 1.138921
>02:29:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 196
>02:29:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:29:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 196
>02:29:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0q4p4xm4.yaml with timeout of 600
>02:29:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-196-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:29:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-196-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-196-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:29:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-196-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:29:29Z with timeout of 600
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:29:29Z with timeout of 600
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:29:29Z with timeout of 600
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:29:29Z with timeout of 600
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:29:29Z with timeout of 600
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:29:29Z with timeout of 600
>02:29:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 196 are : Creation time is 1.206726 , Creation speed 5.966557445517872, Csi creation time is 1.168383
>02:29:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 197
>02:29:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:29:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 197
>02:29:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapj242af_f.yaml with timeout of 600
>02:29:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:29:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-197-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:29:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-197-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-197-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:30:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-197-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:29:54Z with timeout of 600
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:29:54Z with timeout of 600
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:29:54Z with timeout of 600
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:29:54Z with timeout of 600
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:30:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:30:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:29:54Z with timeout of 600
>02:30:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:29:54Z with timeout of 600
>02:30:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 197 are : Creation time is 1.689865 , Creation speed 4.260695380991973, Csi creation time is 1.649159
>02:30:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 198
>02:30:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:30:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 198
>02:30:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7r7jz7h0.yaml with timeout of 600
>02:30:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-198-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:30:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-198-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-198-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:30:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-198-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:30:18Z with timeout of 600
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:30:18Z with timeout of 600
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:30:18Z with timeout of 600
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:30:18Z with timeout of 600
>02:30:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:30:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:30:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:30:18Z with timeout of 600
>02:30:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:30:18Z with timeout of 600
>02:30:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 198 are : Creation time is 1.61978 , Creation speed 4.445048092950895, Csi creation time is 1.579614
>02:30:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 199
>02:30:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:30:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 199
>02:30:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapicghwizr.yaml with timeout of 600
>02:30:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-199-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:30:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-199-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-199-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:30:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-199-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:30:42Z with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:30:42Z with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:30:42Z with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:30:42Z with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:30:42Z with timeout of 600
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:30:42Z with timeout of 600
>02:30:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 199 are : Creation time is 1.492935 , Creation speed 4.8227149875915565, Csi creation time is 1.450555
>02:30:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 200
>02:30:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:31:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 200
>02:31:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapx6iaujvs.yaml with timeout of 600
>02:31:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-200-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:31:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-200-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-200-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:31:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-200-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:31:07Z with timeout of 600
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:31:07Z with timeout of 600
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:31:07Z with timeout of 600
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:31:07Z with timeout of 600
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:31:07Z with timeout of 600
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:31:07Z with timeout of 600
>02:31:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 200 are : Creation time is 2.076973 , Creation speed 3.466583340274524, Csi creation time is 2.039719
>02:31:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 201
>02:31:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:31:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 201
>02:31:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapikv5eukh.yaml with timeout of 600
>02:31:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-201-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:31:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-201-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-201-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:31:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-201-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:31:31Z with timeout of 600
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:31:31Z with timeout of 600
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:31:31Z with timeout of 600
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:31:31Z with timeout of 600
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:31:31Z with timeout of 600
>02:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:31:31Z with timeout of 600
>02:31:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 201 are : Creation time is 1.458452 , Creation speed 4.9367411474632, Csi creation time is 1.418788
>02:31:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 202
>02:31:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:31:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 202
>02:31:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5logbhkb.yaml with timeout of 600
>02:31:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:31:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-202-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:31:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-202-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:32:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-202-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:32:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-202-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:32:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:32:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:31:56Z with timeout of 600
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:31:56Z with timeout of 600
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:31:56Z with timeout of 600
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:31:56Z with timeout of 600
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:31:56Z with timeout of 600
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:31:56Z with timeout of 600
>02:32:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 202 are : Creation time is 1.896373 , Creation speed 3.796721425584524, Csi creation time is 1.858295
>02:32:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 203
>02:32:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:32:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 203
>02:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0g44i_7l.yaml with timeout of 600
>02:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-203-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:32:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-203-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-203-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:32:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-203-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:32:21Z with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:32:21Z with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:32:21Z with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:32:21Z with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:32:21Z with timeout of 600
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:32:21Z with timeout of 600
>02:32:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 203 are : Creation time is 1.832043 , Creation speed 3.9300387600072706, Csi creation time is 1.787726
>02:32:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 204
>02:32:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:32:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 204
>02:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaph3gwc4zh.yaml with timeout of 600
>02:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-204-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:32:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-204-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-204-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:32:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-204-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:32:46Z with timeout of 600
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:32:46Z with timeout of 600
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:32:46Z with timeout of 600
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:32:46Z with timeout of 600
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:32:46Z with timeout of 600
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:32:46Z with timeout of 600
>02:32:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 204 are : Creation time is 1.987874 , Creation speed 3.621959943135229, Csi creation time is 1.951362
>02:32:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 205
>02:32:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:33:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 205
>02:33:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapojcazhi8.yaml with timeout of 600
>02:33:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-205-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:33:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-205-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-205-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:33:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-205-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:33:11Z with timeout of 600
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:33:11Z with timeout of 600
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:33:11Z with timeout of 600
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:33:11Z with timeout of 600
>02:33:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:33:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:33:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:33:11Z with timeout of 600
>02:33:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:33:11Z with timeout of 600
>02:33:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 205 are : Creation time is 1.433638 , Creation speed 5.02218830695057, Csi creation time is 1.393728
>02:33:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 206
>02:33:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:33:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 206
>02:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfjjjte0v.yaml with timeout of 600
>02:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-206-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:33:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-206-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-206-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:33:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-206-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:33:35Z with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:33:35Z with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:33:35Z with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:33:35Z with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:33:35Z with timeout of 600
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:33:35Z with timeout of 600
>02:33:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 206 are : Creation time is 1.917022 , Creation speed 3.7558254417528856, Csi creation time is 1.878965
>02:33:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 207
>02:33:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:34:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 207
>02:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap82n3d9l7.yaml with timeout of 600
>02:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-207-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:34:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-207-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-207-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:34:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-207-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:34:00Z with timeout of 600
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:34:00Z with timeout of 600
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:34:00Z with timeout of 600
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:34:00Z with timeout of 600
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:34:00Z with timeout of 600
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:34:00Z with timeout of 600
>02:34:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 207 are : Creation time is 1.307479 , Creation speed 5.506780606036502, Csi creation time is 1.265452
>02:34:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 208
>02:34:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:34:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 208
>02:34:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3i799exv.yaml with timeout of 600
>02:34:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-208-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:34:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-208-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-208-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:34:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-208-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:34:24Z with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:34:24Z with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:34:24Z with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:34:24Z with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:34:24Z with timeout of 600
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:34:24Z with timeout of 600
>02:34:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 208 are : Creation time is 1.981091 , Creation speed 3.6343610667051642, Csi creation time is 1.942371
>02:34:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 209
>02:34:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:34:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 209
>02:34:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfn0gfq06.yaml with timeout of 600
>02:34:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-209-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:34:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-209-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-209-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:34:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-209-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:34:48Z with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:34:48Z with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:34:48Z with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:34:48Z with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:34:48Z with timeout of 600
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:34:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:34:48Z with timeout of 600
>02:35:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 209 are : Creation time is 1.985307 , Creation speed 3.626643133782332, Csi creation time is 1.944866
>02:35:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 210
>02:35:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:35:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 210
>02:35:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8uve65he.yaml with timeout of 600
>02:35:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-210-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:35:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-210-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:35:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-210-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:35:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-210-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:35:12Z with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:35:12Z with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:35:12Z with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:35:12Z with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:35:12Z with timeout of 600
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:35:12Z with timeout of 600
>02:35:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 210 are : Creation time is 1.545188 , Creation speed 4.659627178052121, Csi creation time is 1.504978
>02:35:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 211
>02:35:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:35:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 211
>02:35:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapw3_jwetl.yaml with timeout of 600
>02:35:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-211-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:35:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-211-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-211-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:35:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-211-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:35:37Z with timeout of 600
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:35:37Z with timeout of 600
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:35:37Z with timeout of 600
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:35:37Z with timeout of 600
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:35:37Z with timeout of 600
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:35:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:35:37Z with timeout of 600
>02:35:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 211 are : Creation time is 2.098502 , Creation speed 3.4310188887120434, Csi creation time is 2.055084
>02:35:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 212
>02:35:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:36:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 212
>02:36:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapu7tj3xro.yaml with timeout of 600
>02:36:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-212-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:36:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-212-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-212-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:36:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-212-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:36:01Z with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:36:01Z with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:36:01Z with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:36:01Z with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:36:01Z with timeout of 600
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:36:01Z with timeout of 600
>02:36:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 212 are : Creation time is 1.520118 , Creation speed 4.736474405276432, Csi creation time is 1.482892
>02:36:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 213
>02:36:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:36:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 213
>02:36:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapupu35flb.yaml with timeout of 600
>02:36:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-213-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:36:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-213-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-213-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:36:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-213-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:36:26Z with timeout of 600
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:36:26Z with timeout of 600
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:36:26Z with timeout of 600
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:36:26Z with timeout of 600
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:36:26Z with timeout of 600
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:36:26Z with timeout of 600
>02:36:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 213 are : Creation time is 1.981618 , Creation speed 3.6333945291171155, Csi creation time is 1.939762
>02:36:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 214
>02:36:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:36:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 214
>02:36:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8v9rxmcf.yaml with timeout of 600
>02:36:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:36:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-214-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:36:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-214-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-214-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:37:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-214-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:36:51Z with timeout of 600
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:36:51Z with timeout of 600
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:36:51Z with timeout of 600
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:36:51Z with timeout of 600
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:36:51Z with timeout of 600
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:36:51Z with timeout of 600
>02:37:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 214 are : Creation time is 2.023037 , Creation speed 3.5590055940647654, Csi creation time is 1.986227
>02:37:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 215
>02:37:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:37:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 215
>02:37:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5rgnf1ki.yaml with timeout of 600
>02:37:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-215-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:37:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-215-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:37:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-215-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:37:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-215-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:37:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:37:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:37:15Z with timeout of 600
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:37:15Z with timeout of 600
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:37:15Z with timeout of 600
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:37:15Z with timeout of 600
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:37:15Z with timeout of 600
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:37:15Z with timeout of 600
>02:37:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 215 are : Creation time is 1.761235 , Creation speed 4.08804049431223, Csi creation time is 1.722788
>02:37:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 216
>02:37:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:37:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 216
>02:37:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapqj6rtne1.yaml with timeout of 600
>02:37:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-216-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:37:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-216-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-216-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:37:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-216-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:37:39Z with timeout of 600
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:37:39Z with timeout of 600
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:37:39Z with timeout of 600
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:37:39Z with timeout of 600
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:37:39Z with timeout of 600
>02:37:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:37:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:37:39Z with timeout of 600
>02:37:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 216 are : Creation time is 1.389877 , Creation speed 5.180314517039997, Csi creation time is 1.354798
>02:37:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 217
>02:37:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:38:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 217
>02:38:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4usf96xa.yaml with timeout of 600
>02:38:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-217-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:38:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-217-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:38:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-217-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:38:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-217-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:38:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:38:04Z with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:38:04Z with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:38:04Z with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:38:04Z with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:38:04Z with timeout of 600
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:38:04Z with timeout of 600
>02:38:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 217 are : Creation time is 1.69453 , Creation speed 4.24896578992405, Csi creation time is 1.653439
>02:38:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 218
>02:38:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:38:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 218
>02:38:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaproctzpuq.yaml with timeout of 600
>02:38:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-218-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:38:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-218-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-218-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:38:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-218-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:38:29Z with timeout of 600
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:38:29Z with timeout of 600
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:38:29Z with timeout of 600
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:38:29Z with timeout of 600
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:38:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:38:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:38:29Z with timeout of 600
>02:38:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:38:29Z with timeout of 600
>02:38:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 218 are : Creation time is 1.256033 , Creation speed 5.732333465760852, Csi creation time is 1.21186
>02:38:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 219
>02:38:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:38:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 219
>02:38:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvdb23d7r.yaml with timeout of 600
>02:38:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:38:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-219-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:38:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-219-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-219-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:39:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-219-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:38:53Z with timeout of 600
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:38:53Z with timeout of 600
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:38:53Z with timeout of 600
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:38:53Z with timeout of 600
>02:39:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:39:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:39:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:38:53Z with timeout of 600
>02:39:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:38:53Z with timeout of 600
>02:39:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 219 are : Creation time is 1.228568 , Creation speed 5.860481471111082, Csi creation time is 1.18443
>02:39:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 220
>02:39:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:39:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 220
>02:39:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapo_owizcr.yaml with timeout of 600
>02:39:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-220-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:39:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-220-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-220-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:39:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-220-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:39:18Z with timeout of 600
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:39:18Z with timeout of 600
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:39:18Z with timeout of 600
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:39:18Z with timeout of 600
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:39:18Z with timeout of 600
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:39:18Z with timeout of 600
>02:39:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 220 are : Creation time is 1.863703 , Creation speed 3.8632764984549577, Csi creation time is 1.829351
>02:39:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 221
>02:39:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:39:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 221
>02:39:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapkopgzqgx.yaml with timeout of 600
>02:39:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-221-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:39:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-221-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-221-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:39:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-221-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:39:44Z with timeout of 600
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:39:44Z with timeout of 600
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:39:44Z with timeout of 600
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:39:44Z with timeout of 600
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:39:44Z with timeout of 600
>02:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:39:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:39:44Z with timeout of 600
>02:39:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 221 are : Creation time is 1.490281 , Creation speed 4.831303626631488, Csi creation time is 1.446966
>02:39:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 222
>02:39:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:40:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 222
>02:40:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaplghwu2p4.yaml with timeout of 600
>02:40:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-222-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:40:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-222-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-222-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:40:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-222-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:40:10Z with timeout of 600
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:40:10Z with timeout of 600
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:40:10Z with timeout of 600
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:40:10Z with timeout of 600
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:40:10Z with timeout of 600
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:40:10Z with timeout of 600
>02:40:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 222 are : Creation time is 2.018572 , Creation speed 3.5668779711598106, Csi creation time is 1.983213
>02:40:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 223
>02:40:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:40:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 223
>02:40:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_9o0l6mq.yaml with timeout of 600
>02:40:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-223-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:40:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-223-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-223-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:40:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-223-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:40:35Z with timeout of 600
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:40:35Z with timeout of 600
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:40:35Z with timeout of 600
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:40:35Z with timeout of 600
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:40:35Z with timeout of 600
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:40:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:40:35Z with timeout of 600
>02:40:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 223 are : Creation time is 1.928022 , Creation speed 3.7343972216084675, Csi creation time is 1.886582
>02:40:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 224
>02:40:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:41:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 224
>02:41:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvl15j9o3.yaml with timeout of 600
>02:41:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-224-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:41:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-224-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:41:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-224-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:41:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-224-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:41:00Z with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:41:00Z with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:41:00Z with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:41:00Z with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:41:00Z with timeout of 600
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:41:00Z with timeout of 600
>02:41:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 224 are : Creation time is 1.573103 , Creation speed 4.576941242881109, Csi creation time is 1.538664
>02:41:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 225
>02:41:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:41:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 225
>02:41:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapr61vkakj.yaml with timeout of 600
>02:41:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-225-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:41:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-225-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-225-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:41:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-225-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:41:25Z with timeout of 600
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:41:25Z with timeout of 600
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:41:25Z with timeout of 600
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:41:25Z with timeout of 600
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:41:25Z with timeout of 600
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:41:25Z with timeout of 600
>02:41:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 225 are : Creation time is 1.940903 , Creation speed 3.7096135149463936, Csi creation time is 1.899348
>02:41:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 226
>02:41:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:41:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 226
>02:41:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap75evhei9.yaml with timeout of 600
>02:41:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:41:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-226-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:41:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-226-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:42:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-226-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:42:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-226-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:41:50Z with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:41:50Z with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:41:50Z with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:41:50Z with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:41:50Z with timeout of 600
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:41:50Z with timeout of 600
>02:42:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 226 are : Creation time is 1.597248 , Creation speed 4.507753335737468, Csi creation time is 1.561364
>02:42:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 227
>02:42:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:42:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 227
>02:42:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8vo6sxnp.yaml with timeout of 600
>02:42:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-227-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:42:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-227-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-227-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:42:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-227-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:42:15Z with timeout of 600
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:42:15Z with timeout of 600
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:42:15Z with timeout of 600
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:42:15Z with timeout of 600
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:42:15Z with timeout of 600
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:42:15Z with timeout of 600
>02:42:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 227 are : Creation time is 2.10923 , Creation speed 3.4135679845251583, Csi creation time is 2.067976
>02:42:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 228
>02:42:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:42:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 228
>02:42:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprij5e5b6.yaml with timeout of 600
>02:42:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-228-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:42:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-228-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-228-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:42:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-228-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:42:40Z with timeout of 600
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:42:40Z with timeout of 600
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:42:40Z with timeout of 600
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:42:40Z with timeout of 600
>02:42:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:42:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:42:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:42:40Z with timeout of 600
>02:42:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:42:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:42:40Z with timeout of 600
>02:42:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 228 are : Creation time is 2.162066 , Creation speed 3.3301481083371187, Csi creation time is 2.120774
>02:42:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 229
>02:42:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:43:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 229
>02:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphxe3j2j9.yaml with timeout of 600
>02:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-229-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:43:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-229-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:43:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-229-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:43:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-229-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:43:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:43:04Z with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:43:04Z with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:43:04Z with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:43:04Z with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:43:04Z with timeout of 600
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:43:04Z with timeout of 600
>02:43:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 229 are : Creation time is 1.849261 , Creation speed 3.893447166192333, Csi creation time is 1.800899
>02:43:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 230
>02:43:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:43:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 230
>02:43:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5lqmgrlu.yaml with timeout of 600
>02:43:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-230-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:43:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-230-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-230-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:43:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-230-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:43:28Z with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:43:28Z with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:43:28Z with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:43:28Z with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:43:28Z with timeout of 600
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:43:28Z with timeout of 600
>02:43:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 230 are : Creation time is 1.512872 , Creation speed 4.759160061128767, Csi creation time is 1.478689
>02:43:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 231
>02:43:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:43:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 231
>02:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5b74imay.yaml with timeout of 600
>02:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-231-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:43:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-231-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-231-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:44:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-231-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:43:52Z with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:43:52Z with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:43:52Z with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:43:52Z with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:43:52Z with timeout of 600
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:43:52Z with timeout of 600
>02:44:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 231 are : Creation time is 1.66597 , Creation speed 4.321806515123321, Csi creation time is 1.627489
>02:44:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 232
>02:44:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:44:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 232
>02:44:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5yniug69.yaml with timeout of 600
>02:44:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-232-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:44:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-232-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:44:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-232-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:44:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-232-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:44:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:44:16Z with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:44:16Z with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:44:16Z with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:44:16Z with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:44:16Z with timeout of 600
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:44:16Z with timeout of 600
>02:44:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 232 are : Creation time is 1.872792 , Creation speed 3.8445273153665758, Csi creation time is 1.832303
>02:44:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 233
>02:44:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:44:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 233
>02:44:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapcg91tuyr.yaml with timeout of 600
>02:44:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-233-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:44:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-233-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-233-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:44:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-233-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:44:40Z with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:44:40Z with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:44:40Z with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:44:40Z with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:44:40Z with timeout of 600
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:44:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:44:40Z with timeout of 600
>02:44:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 233 are : Creation time is 1.512541 , Creation speed 4.760201541644161, Csi creation time is 1.474716
>02:44:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 234
>02:44:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:45:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 234
>02:45:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3s7wk5gm.yaml with timeout of 600
>02:45:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-234-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:45:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-234-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-234-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:45:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-234-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:45:05Z with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:45:05Z with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:45:05Z with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:45:05Z with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:45:05Z with timeout of 600
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:45:05Z with timeout of 600
>02:45:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 234 are : Creation time is 1.674613 , Creation speed 4.299500839895546, Csi creation time is 1.641098
>02:45:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 235
>02:45:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:45:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 235
>02:45:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphvn5e138.yaml with timeout of 600
>02:45:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-235-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:45:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-235-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-235-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:45:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-235-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:45:30Z with timeout of 600
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:45:30Z with timeout of 600
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:45:30Z with timeout of 600
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:45:30Z with timeout of 600
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:45:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:45:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:45:30Z with timeout of 600
>02:45:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:45:30Z with timeout of 600
>02:45:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 235 are : Creation time is 1.366737 , Creation speed 5.2680215725483395, Csi creation time is 1.326711
>02:45:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 236
>02:45:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:45:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 236
>02:45:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapw3dqe1cg.yaml with timeout of 600
>02:45:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:45:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-236-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:45:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-236-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-236-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:46:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-236-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:45:54Z with timeout of 600
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:45:54Z with timeout of 600
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:45:54Z with timeout of 600
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:45:54Z with timeout of 600
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:46:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:46:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:45:54Z with timeout of 600
>02:46:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:45:54Z with timeout of 600
>02:46:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 236 are : Creation time is 1.464778 , Creation speed 4.915420630293465, Csi creation time is 1.431255
>02:46:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 237
>02:46:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:46:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 237
>02:46:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvqtr_mle.yaml with timeout of 600
>02:46:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-237-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:46:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-237-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-237-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:46:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-237-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:46:18Z with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:46:18Z with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:46:18Z with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:46:18Z with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:46:18Z with timeout of 600
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:46:18Z with timeout of 600
>02:46:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 237 are : Creation time is 1.591895 , Creation speed 4.522911372923465, Csi creation time is 1.545961
>02:46:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 238
>02:46:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:46:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 238
>02:46:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap81wtgyfp.yaml with timeout of 600
>02:46:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-238-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:46:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-238-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-238-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:46:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-238-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:46:43Z with timeout of 600
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:46:43Z with timeout of 600
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:46:43Z with timeout of 600
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:46:43Z with timeout of 600
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:46:43Z with timeout of 600
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:46:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:46:43Z with timeout of 600
>02:46:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 238 are : Creation time is 1.216196 , Creation speed 5.920098405191268, Csi creation time is 1.178744
>02:46:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 239
>02:46:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:47:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 239
>02:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapep28zjmz.yaml with timeout of 600
>02:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-239-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:47:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-239-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:47:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-239-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:47:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-239-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:47:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:47:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:47:07Z with timeout of 600
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:47:07Z with timeout of 600
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:47:07Z with timeout of 600
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:47:07Z with timeout of 600
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:47:07Z with timeout of 600
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:47:07Z with timeout of 600
>02:47:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 239 are : Creation time is 1.873978 , Creation speed 3.8420941974772385, Csi creation time is 1.833483
>02:47:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 240
>02:47:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:47:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 240
>02:47:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjow4q0wk.yaml with timeout of 600
>02:47:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-240-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:47:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-240-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-240-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:47:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-240-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:47:32Z with timeout of 600
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:47:32Z with timeout of 600
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:47:32Z with timeout of 600
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:47:32Z with timeout of 600
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:47:32Z with timeout of 600
>02:47:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:47:32Z with timeout of 600
>02:47:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 240 are : Creation time is 1.554447 , Creation speed 4.631872299280709, Csi creation time is 1.516343
>02:47:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 241
>02:47:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:47:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 241
>02:47:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5ux1wz6h.yaml with timeout of 600
>02:47:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:47:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-241-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:47:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-241-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:48:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-241-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:48:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-241-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:48:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:48:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:47:56Z with timeout of 600
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:47:56Z with timeout of 600
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:47:56Z with timeout of 600
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:47:56Z with timeout of 600
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:47:56Z with timeout of 600
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:47:56Z with timeout of 600
>02:48:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 241 are : Creation time is 2.038866 , Creation speed 3.5313747936352855, Csi creation time is 1.995644
>02:48:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 242
>02:48:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:48:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 242
>02:48:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap82v0w_d4.yaml with timeout of 600
>02:48:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-242-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:48:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-242-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-242-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:48:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-242-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:48:21Z with timeout of 600
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:48:21Z with timeout of 600
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:48:21Z with timeout of 600
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:48:21Z with timeout of 600
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:48:21Z with timeout of 600
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:48:21Z with timeout of 600
>02:48:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 242 are : Creation time is 1.184291 , Creation speed 6.079586858297496, Csi creation time is 1.139959
>02:48:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 243
>02:48:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:48:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 243
>02:48:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdfm4u4g_.yaml with timeout of 600
>02:48:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-243-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:48:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-243-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-243-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:48:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-243-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:48:45Z with timeout of 600
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:48:45Z with timeout of 600
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:48:45Z with timeout of 600
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:48:45Z with timeout of 600
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:48:45Z with timeout of 600
>02:48:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:48:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:48:45Z with timeout of 600
>02:48:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 243 are : Creation time is 1.692691 , Creation speed 4.253582018218328, Csi creation time is 1.650792
>02:48:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 244
>02:48:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:49:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 244
>02:49:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapqcf_njtf.yaml with timeout of 600
>02:49:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-244-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:49:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-244-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-244-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:49:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-244-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:49:12Z with timeout of 600
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:49:12Z with timeout of 600
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:49:12Z with timeout of 600
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:49:12Z with timeout of 600
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:49:12Z with timeout of 600
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:49:12Z with timeout of 600
>02:49:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 244 are : Creation time is 2.098356 , Creation speed 3.4312576131028294, Csi creation time is 2.063007
>02:49:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 245
>02:49:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:49:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 245
>02:49:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapciu0jtg9.yaml with timeout of 600
>02:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-245-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:49:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-245-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-245-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:49:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-245-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:49:36Z with timeout of 600
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:49:36Z with timeout of 600
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:49:36Z with timeout of 600
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:49:36Z with timeout of 600
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:49:36Z with timeout of 600
>02:49:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:49:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:49:36Z with timeout of 600
>02:49:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 245 are : Creation time is 1.641591 , Creation speed 4.385988958272797, Csi creation time is 1.590901
>02:49:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 246
>02:49:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:50:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 246
>02:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdyedlqwr.yaml with timeout of 600
>02:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-246-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:50:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-246-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-246-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:50:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-246-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:50:01Z with timeout of 600
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:50:01Z with timeout of 600
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:50:01Z with timeout of 600
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:50:01Z with timeout of 600
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:50:01Z with timeout of 600
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:50:01Z with timeout of 600
>02:50:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 246 are : Creation time is 2.150931 , Creation speed 3.347387712576554, Csi creation time is 2.108429
>02:50:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 247
>02:50:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:50:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 247
>02:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4q4apn2_.yaml with timeout of 600
>02:50:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-247-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:50:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-247-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-247-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:50:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-247-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:50:25Z with timeout of 600
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:50:25Z with timeout of 600
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:50:25Z with timeout of 600
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:50:25Z with timeout of 600
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:50:25Z with timeout of 600
>02:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:50:25Z with timeout of 600
>02:50:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 247 are : Creation time is 1.662831 , Creation speed 4.329964981408213, Csi creation time is 1.62027
>02:50:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 248
>02:50:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:50:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 248
>02:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap9ji0c1s7.yaml with timeout of 600
>02:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:50:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-248-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:50:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-248-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-248-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:51:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-248-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:50:50Z with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:50:50Z with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:50:50Z with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:50:50Z with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:50:50Z with timeout of 600
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:50:50Z with timeout of 600
>02:51:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 248 are : Creation time is 1.66203 , Creation speed 4.332051768018628, Csi creation time is 1.619398
>02:51:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 249
>02:51:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:51:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 249
>02:51:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapnpquofvf.yaml with timeout of 600
>02:51:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-249-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:51:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-249-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-249-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:51:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-249-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:51:15Z with timeout of 600
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:51:15Z with timeout of 600
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:51:15Z with timeout of 600
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:51:15Z with timeout of 600
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:51:15Z with timeout of 600
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:51:15Z with timeout of 600
>02:51:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 249 are : Creation time is 1.367896 , Creation speed 5.263558048272676, Csi creation time is 1.312331
>02:51:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 250
>02:51:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:51:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 250
>02:51:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphy702lny.yaml with timeout of 600
>02:51:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-250-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:51:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-250-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:51:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-250-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:51:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-250-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:51:39Z with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:51:39Z with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:51:39Z with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:51:39Z with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:51:39Z with timeout of 600
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:51:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:51:39Z with timeout of 600
>02:51:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 250 are : Creation time is 1.723292 , Creation speed 4.178049918411969, Csi creation time is 1.688487
>02:51:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 251
>02:51:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:52:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 251
>02:52:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvmp32yxg.yaml with timeout of 600
>02:52:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-251-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:52:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-251-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-251-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:52:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-251-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:52:03Z with timeout of 600
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:52:03Z with timeout of 600
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:52:03Z with timeout of 600
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:52:03Z with timeout of 600
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:52:03Z with timeout of 600
>02:52:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:52:03Z with timeout of 600
>02:52:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 251 are : Creation time is 1.757306 , Creation speed 4.0971805707144915, Csi creation time is 1.720619
>02:52:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 252
>02:52:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:52:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 252
>02:52:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjepklchb.yaml with timeout of 600
>02:52:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-252-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:52:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-252-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-252-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:52:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-252-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:52:28Z with timeout of 600
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:52:28Z with timeout of 600
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:52:28Z with timeout of 600
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:52:28Z with timeout of 600
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:52:28Z with timeout of 600
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:52:28Z with timeout of 600
>02:52:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 252 are : Creation time is 1.8847 , Creation speed 3.820236642436462, Csi creation time is 1.846317
>02:52:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 253
>02:52:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:52:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 253
>02:52:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap9j4w_x45.yaml with timeout of 600
>02:52:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:52:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-253-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:52:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-253-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:53:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-253-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:53:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-253-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:53:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:53:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:52:52Z with timeout of 600
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:52:52Z with timeout of 600
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:52:52Z with timeout of 600
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:52:52Z with timeout of 600
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:52:52Z with timeout of 600
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:52:52Z with timeout of 600
>02:53:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 253 are : Creation time is 2.123359 , Creation speed 3.390853831123234, Csi creation time is 2.081257
>02:53:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 254
>02:53:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:53:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 254
>02:53:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphk9tastw.yaml with timeout of 600
>02:53:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-254-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:53:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-254-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-254-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:53:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-254-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:53:16Z with timeout of 600
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:53:16Z with timeout of 600
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:53:16Z with timeout of 600
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:53:16Z with timeout of 600
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:53:16Z with timeout of 600
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:53:16Z with timeout of 600
>02:53:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 254 are : Creation time is 2.294648 , Creation speed 3.137736158225575, Csi creation time is 2.255773
>02:53:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 255
>02:53:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:53:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 255
>02:53:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsen6qyjr.yaml with timeout of 600
>02:53:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-255-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:53:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-255-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:53:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-255-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:53:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-255-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:53:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:53:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:53:40Z with timeout of 600
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:53:40Z with timeout of 600
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:53:40Z with timeout of 600
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:53:40Z with timeout of 600
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:53:40Z with timeout of 600
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:53:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:53:40Z with timeout of 600
>02:53:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 255 are : Creation time is 1.927426 , Creation speed 3.7355519744986316, Csi creation time is 1.88822
>02:53:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 256
>02:53:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:54:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 256
>02:54:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8qgt7_x1.yaml with timeout of 600
>02:54:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-256-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:54:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-256-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:54:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-256-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:54:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-256-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:54:06Z with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:54:06Z with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:54:06Z with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:54:06Z with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:54:06Z with timeout of 600
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:54:06Z with timeout of 600
>02:54:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 256 are : Creation time is 2.005475 , Creation speed 3.590171904411673, Csi creation time is 1.972502
>02:54:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 257
>02:54:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:54:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 257
>02:54:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsdoq4avv.yaml with timeout of 600
>02:54:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-257-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:54:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-257-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-257-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:54:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-257-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:54:32Z with timeout of 600
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:54:32Z with timeout of 600
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:54:32Z with timeout of 600
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:54:32Z with timeout of 600
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:54:32Z with timeout of 600
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:54:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:54:32Z with timeout of 600
>02:54:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 257 are : Creation time is 2.146767 , Creation speed 3.3538805096221433, Csi creation time is 2.109037
>02:54:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 258
>02:54:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:55:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 258
>02:55:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap313y40oo.yaml with timeout of 600
>02:55:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-258-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:55:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-258-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-258-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:55:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-258-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:55:00Z with timeout of 600
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:55:00Z with timeout of 600
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:55:00Z with timeout of 600
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:55:00Z with timeout of 600
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:55:00Z with timeout of 600
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:55:00Z with timeout of 600
>02:55:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 258 are : Creation time is 2.193522 , Creation speed 3.2823924264265414, Csi creation time is 2.154594
>02:55:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 259
>02:55:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:55:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 259
>02:55:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapza3ht2ml.yaml with timeout of 600
>02:55:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-259-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:55:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-259-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-259-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:55:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-259-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:55:24Z with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:55:24Z with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:55:24Z with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:55:24Z with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:55:24Z with timeout of 600
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:55:24Z with timeout of 600
>02:55:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 259 are : Creation time is 1.9419 , Creation speed 3.7077089448478295, Csi creation time is 1.905394
>02:55:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 260
>02:55:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:55:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 260
>02:55:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapymth98cu.yaml with timeout of 600
>02:55:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-260-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:55:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-260-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-260-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:55:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-260-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:55:49Z with timeout of 600
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:55:49Z with timeout of 600
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:55:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:55:49Z with timeout of 600
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:55:49Z with timeout of 600
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:55:49Z with timeout of 600
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:55:49Z with timeout of 600
>02:56:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 260 are : Creation time is 2.153945 , Creation speed 3.3427037366320866, Csi creation time is 2.11432
>02:56:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 261
>02:56:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:56:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 261
>02:56:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapxupzufc8.yaml with timeout of 600
>02:56:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-261-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:56:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-261-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-261-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:56:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-261-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:56:13Z with timeout of 600
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:56:13Z with timeout of 600
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:56:13Z with timeout of 600
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:56:13Z with timeout of 600
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:56:13Z with timeout of 600
>02:56:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:56:13Z with timeout of 600
>02:56:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 261 are : Creation time is 1.908055 , Creation speed 3.7734761314532337, Csi creation time is 1.870276
>02:56:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 262
>02:56:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:56:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 262
>02:56:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapc_tv8ksd.yaml with timeout of 600
>02:56:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-262-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:56:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-262-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-262-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:56:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-262-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:56:38Z with timeout of 600
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:56:38Z with timeout of 600
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:56:38Z with timeout of 600
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:56:38Z with timeout of 600
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:56:38Z with timeout of 600
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:56:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:56:38Z with timeout of 600
>02:56:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 262 are : Creation time is 2.155182 , Creation speed 3.3407851401876965, Csi creation time is 2.1149
>02:56:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 263
>02:56:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:57:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 263
>02:57:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapd9cwmmj9.yaml with timeout of 600
>02:57:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-263-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:57:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-263-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-263-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:57:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-263-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:57:02Z with timeout of 600
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:57:02Z with timeout of 600
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:57:02Z with timeout of 600
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:57:02Z with timeout of 600
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:57:02Z with timeout of 600
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:57:02Z with timeout of 600
>02:57:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 263 are : Creation time is 2.308269 , Creation speed 3.119220506795352, Csi creation time is 2.267564
>02:57:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 264
>02:57:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:57:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 264
>02:57:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap472rpf2r.yaml with timeout of 600
>02:57:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-264-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:57:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-264-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:57:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-264-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:57:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-264-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:57:26Z with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:57:26Z with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:57:26Z with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:57:26Z with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:57:26Z with timeout of 600
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:57:26Z with timeout of 600
>02:57:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 264 are : Creation time is 2.727063 , Creation speed 2.6402030316131313, Csi creation time is 2.686216
>02:57:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 265
>02:57:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:57:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 265
>02:57:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap102l4j7p.yaml with timeout of 600
>02:57:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:57:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-265-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:57:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-265-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-265-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:58:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-265-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:57:51Z with timeout of 600
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:57:51Z with timeout of 600
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:57:51Z with timeout of 600
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:57:51Z with timeout of 600
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:58:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:58:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:57:51Z with timeout of 600
>02:58:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:57:51Z with timeout of 600
>02:58:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 265 are : Creation time is 2.679882 , Creation speed 2.686685458538846, Csi creation time is 2.639164
>02:58:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 266
>02:58:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:58:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 266
>02:58:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7rvnlxo7.yaml with timeout of 600
>02:58:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-266-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:58:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-266-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-266-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:58:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-266-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:58:15Z with timeout of 600
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:58:15Z with timeout of 600
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:58:15Z with timeout of 600
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:58:15Z with timeout of 600
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:58:15Z with timeout of 600
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:58:15Z with timeout of 600
>02:58:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 266 are : Creation time is 2.473089 , Creation speed 2.911338815546064, Csi creation time is 2.432124
>02:58:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 267
>02:58:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:58:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 267
>02:58:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvy1ckhak.yaml with timeout of 600
>02:58:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-267-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:58:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-267-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-267-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:58:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-267-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:58:39Z with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:58:39Z with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:58:39Z with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:58:39Z with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:58:39Z with timeout of 600
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:58:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:58:39Z with timeout of 600
>02:58:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 267 are : Creation time is 2.683123 , Creation speed 2.683440155371185, Csi creation time is 2.646891
>02:58:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 268
>02:58:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:59:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 268
>02:59:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap9bcg25th.yaml with timeout of 600
>02:59:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-268-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:59:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-268-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-268-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:59:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-268-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:59:04Z with timeout of 600
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:59:04Z with timeout of 600
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:59:04Z with timeout of 600
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:59:04Z with timeout of 600
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:59:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:59:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:59:04Z with timeout of 600
>02:59:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:59:04Z with timeout of 600
>02:59:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 268 are : Creation time is 2.777473 , Creation speed 2.59228442544716, Csi creation time is 2.737067
>02:59:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 269
>02:59:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:59:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 269
>02:59:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapr2a744ri.yaml with timeout of 600
>02:59:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-269-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:59:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-269-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-269-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:59:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-269-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:59:28Z with timeout of 600
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:59:28Z with timeout of 600
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:59:28Z with timeout of 600
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:59:28Z with timeout of 600
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:59:28Z with timeout of 600
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:59:28Z with timeout of 600
>02:59:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 269 are : Creation time is 3.115987 , Creation speed 2.310664325621384, Csi creation time is 3.079131
>02:59:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 270
>02:59:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>02:59:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 270
>02:59:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4yky_r95.yaml with timeout of 600
>02:59:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>02:59:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-270-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>02:59:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-270-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-270-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:00:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-270-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:59:52Z with timeout of 600
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:59:52Z with timeout of 600
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T10:59:52Z with timeout of 600
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T10:59:52Z with timeout of 600
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T10:59:52Z with timeout of 600
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T10:59:52Z with timeout of 600
>03:00:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 270 are : Creation time is 2.352342 , Creation speed 3.060779427481208, Csi creation time is 2.313186
>03:00:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 271
>03:00:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:00:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 271
>03:00:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprmeff2m_.yaml with timeout of 600
>03:00:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-271-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:00:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-271-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:00:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-271-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:00:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-271-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:00:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:00:16Z with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:00:16Z with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:00:16Z with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:00:16Z with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:00:16Z with timeout of 600
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:00:16Z with timeout of 600
>03:00:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 271 are : Creation time is 2.909978 , Creation speed 2.4742455097598675, Csi creation time is 2.872791
>03:00:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 272
>03:00:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:00:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 272
>03:00:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapgnfvwi94.yaml with timeout of 600
>03:00:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-272-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:00:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-272-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:00:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-272-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:00:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-272-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:00:41Z with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:00:41Z with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:00:41Z with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:00:41Z with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:00:41Z with timeout of 600
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:00:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:00:41Z with timeout of 600
>03:00:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 272 are : Creation time is 3.064025 , Creation speed 2.3498502786367603, Csi creation time is 3.022859
>03:00:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 273
>03:00:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:01:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 273
>03:01:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1wm83_vf.yaml with timeout of 600
>03:01:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-273-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:01:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-273-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-273-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:01:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-273-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:01:07Z with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:01:07Z with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:01:07Z with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:01:07Z with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:01:07Z with timeout of 600
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:01:07Z with timeout of 600
>03:01:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 273 are : Creation time is 2.750786 , Creation speed 2.617433708038357, Csi creation time is 2.715042
>03:01:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 274
>03:01:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:01:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 274
>03:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapp_aq2ggl.yaml with timeout of 600
>03:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-274-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:01:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-274-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-274-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:01:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-274-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:01:32Z with timeout of 600
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:01:32Z with timeout of 600
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:01:32Z with timeout of 600
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:01:32Z with timeout of 600
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:01:32Z with timeout of 600
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:01:32Z with timeout of 600
>03:01:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 274 are : Creation time is 2.647947 , Creation speed 2.71908765545534, Csi creation time is 2.598038
>03:01:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 275
>03:01:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:01:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 275
>03:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap49ehgmw7.yaml with timeout of 600
>03:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:01:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-275-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:01:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-275-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-275-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:02:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-275-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:01:56Z with timeout of 600
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:01:56Z with timeout of 600
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:01:56Z with timeout of 600
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:01:56Z with timeout of 600
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:01:56Z with timeout of 600
>03:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:01:56Z with timeout of 600
>03:02:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 275 are : Creation time is 3.507791 , Creation speed 2.0525738278021697, Csi creation time is 3.472098
>03:02:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 276
>03:02:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:02:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 276
>03:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7xm5zahz.yaml with timeout of 600
>03:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-276-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:02:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-276-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-276-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:02:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-276-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:02:20Z with timeout of 600
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:02:20Z with timeout of 600
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:02:20Z with timeout of 600
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:02:20Z with timeout of 600
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:02:20Z with timeout of 600
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:02:20Z with timeout of 600
>03:02:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 276 are : Creation time is 3.29446 , Creation speed 2.1854871511567904, Csi creation time is 3.251774
>03:02:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 277
>03:02:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:02:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 277
>03:02:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4xofsg9m.yaml with timeout of 600
>03:02:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-277-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:02:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-277-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-277-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:02:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-277-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:02:45Z with timeout of 600
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:02:45Z with timeout of 600
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:02:45Z with timeout of 600
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:02:45Z with timeout of 600
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:02:45Z with timeout of 600
>03:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:02:45Z with timeout of 600
>03:02:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 277 are : Creation time is 2.865068 , Creation speed 2.513029359163552, Csi creation time is 2.828455
>03:02:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 278
>03:02:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:03:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 278
>03:03:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4r76zl06.yaml with timeout of 600
>03:03:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-278-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:03:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-278-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-278-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:03:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-278-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:03:09Z with timeout of 600
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:03:09Z with timeout of 600
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:03:09Z with timeout of 600
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:03:09Z with timeout of 600
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:03:09Z with timeout of 600
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:03:09Z with timeout of 600
>03:03:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 278 are : Creation time is 3.22865 , Creation speed 2.2300342248308116, Csi creation time is 3.187693
>03:03:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 279
>03:03:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:03:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 279
>03:03:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvqd1vdc7.yaml with timeout of 600
>03:03:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-279-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:03:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-279-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-279-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:03:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-279-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:03:34Z with timeout of 600
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:03:34Z with timeout of 600
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:03:34Z with timeout of 600
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:03:34Z with timeout of 600
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:03:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:03:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:03:34Z with timeout of 600
>03:03:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:03:34Z with timeout of 600
>03:03:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 279 are : Creation time is 3.809007 , Creation speed 1.890256436913873, Csi creation time is 3.770278
>03:03:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 280
>03:03:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:03:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 280
>03:03:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapool9tes0.yaml with timeout of 600
>03:03:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:03:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-280-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:03:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-280-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-280-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:04:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-280-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:03:59Z with timeout of 600
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:03:59Z with timeout of 600
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:03:59Z with timeout of 600
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:03:59Z with timeout of 600
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:04:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:04:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:03:59Z with timeout of 600
>03:04:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:03:59Z with timeout of 600
>03:04:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 280 are : Creation time is 3.445108 , Creation speed 2.089919967675905, Csi creation time is 3.406048
>03:04:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 281
>03:04:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:04:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 281
>03:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapm5smwy7l.yaml with timeout of 600
>03:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-281-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:04:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-281-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-281-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:04:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-281-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:04:23Z with timeout of 600
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:04:23Z with timeout of 600
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:04:23Z with timeout of 600
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:04:23Z with timeout of 600
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:04:23Z with timeout of 600
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:04:23Z with timeout of 600
>03:04:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 281 are : Creation time is 3.476345 , Creation speed 2.071140810247545, Csi creation time is 3.440023
>03:04:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 282
>03:04:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:04:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 282
>03:04:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapm_jqv3yd.yaml with timeout of 600
>03:04:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-282-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:04:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-282-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-282-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:04:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-282-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:04:48Z with timeout of 600
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:04:48Z with timeout of 600
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:04:48Z with timeout of 600
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:04:48Z with timeout of 600
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:04:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:04:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:04:48Z with timeout of 600
>03:04:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:04:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:04:48Z with timeout of 600
>03:04:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 282 are : Creation time is 3.514927 , Creation speed 2.0484066952172832, Csi creation time is 3.475744
>03:04:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 283
>03:04:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:05:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 283
>03:05:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprlpubur5.yaml with timeout of 600
>03:05:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-283-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:05:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-283-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:05:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-283-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:05:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-283-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:05:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:05:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:05:13Z with timeout of 600
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:05:13Z with timeout of 600
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:05:13Z with timeout of 600
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:05:13Z with timeout of 600
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:05:13Z with timeout of 600
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:05:13Z with timeout of 600
>03:05:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 283 are : Creation time is 3.409219 , Creation speed 2.111920648101515, Csi creation time is 3.367856
>03:05:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 284
>03:05:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:05:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 284
>03:05:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0ec9adba.yaml with timeout of 600
>03:05:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-284-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:05:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-284-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-284-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:05:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-284-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:05:39Z with timeout of 600
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:05:39Z with timeout of 600
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:05:39Z with timeout of 600
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:05:39Z with timeout of 600
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:05:39Z with timeout of 600
>03:05:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:05:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:05:39Z with timeout of 600
>03:05:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 284 are : Creation time is 3.85972 , Creation speed 1.8654202895546828, Csi creation time is 3.796993
>03:05:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 285
>03:05:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:06:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 285
>03:06:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap6_4fotrb.yaml with timeout of 600
>03:06:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-285-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:06:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-285-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:06:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-285-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:06:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-285-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:06:05Z with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:06:05Z with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:06:05Z with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:06:05Z with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:06:05Z with timeout of 600
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:06:05Z with timeout of 600
>03:06:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 285 are : Creation time is 3.87474 , Creation speed 1.8581891946298332, Csi creation time is 3.839064
>03:06:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 286
>03:06:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:06:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 286
>03:06:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapd9amtxyq.yaml with timeout of 600
>03:06:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-286-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:06:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-286-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:06:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-286-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:06:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-286-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:06:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:06:30Z with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:06:30Z with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:06:30Z with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:06:30Z with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:06:30Z with timeout of 600
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:06:30Z with timeout of 600
>03:06:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 286 are : Creation time is 4.123012 , Creation speed 1.7462961543648188, Csi creation time is 4.085185
>03:06:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 287
>03:06:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:06:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 287
>03:06:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap9ezee2oa.yaml with timeout of 600
>03:06:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:06:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-287-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:06:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-287-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:07:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-287-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:07:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-287-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:06:54Z with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:06:54Z with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:06:54Z with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:06:54Z with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:06:54Z with timeout of 600
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:06:54Z with timeout of 600
>03:07:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 287 are : Creation time is 4.202567 , Creation speed 1.7132385991704593, Csi creation time is 4.168772
>03:07:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 288
>03:07:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:07:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 288
>03:07:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapamtpspjr.yaml with timeout of 600
>03:07:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-288-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:07:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-288-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-288-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:07:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-288-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:07:19Z with timeout of 600
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:07:19Z with timeout of 600
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:07:19Z with timeout of 600
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:07:19Z with timeout of 600
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:07:19Z with timeout of 600
>03:07:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:07:19Z with timeout of 600
>03:07:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 288 are : Creation time is 3.636702 , Creation speed 1.979815778141844, Csi creation time is 3.597798
>03:07:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 289
>03:07:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:07:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 289
>03:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaplg1ca90v.yaml with timeout of 600
>03:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-289-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:07:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-289-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-289-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:07:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-289-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:07:43Z with timeout of 600
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:07:43Z with timeout of 600
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:07:43Z with timeout of 600
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:07:43Z with timeout of 600
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:07:43Z with timeout of 600
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:07:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:07:43Z with timeout of 600
>03:07:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 289 are : Creation time is 4.51663 , Creation speed 1.5941088820647253, Csi creation time is 4.482023
>03:07:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 290
>03:07:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:08:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 290
>03:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snape5jzmy0s.yaml with timeout of 600
>03:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-290-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:08:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-290-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-290-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:08:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-290-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:08:07Z with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:08:07Z with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:08:07Z with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:08:07Z with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:08:07Z with timeout of 600
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:08:07Z with timeout of 600
>03:08:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 290 are : Creation time is 3.999661 , Creation speed 1.8001525629297084, Csi creation time is 3.956525
>03:08:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 291
>03:08:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:08:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 291
>03:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprmh37nfx.yaml with timeout of 600
>03:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-291-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:08:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-291-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-291-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:08:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-291-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:08:31Z with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:08:31Z with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:08:31Z with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:08:31Z with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:08:31Z with timeout of 600
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:08:31Z with timeout of 600
>03:08:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 291 are : Creation time is 4.268115 , Creation speed 1.6869273672335448, Csi creation time is 4.232942
>03:08:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 292
>03:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:08:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 292
>03:08:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_ns5bmlf.yaml with timeout of 600
>03:08:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:08:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-292-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:08:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-292-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-292-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:09:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-292-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:08:56Z with timeout of 600
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:08:56Z with timeout of 600
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:08:56Z with timeout of 600
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:08:56Z with timeout of 600
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:09:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:09:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:08:56Z with timeout of 600
>03:09:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:08:56Z with timeout of 600
>03:09:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 292 are : Creation time is 4.595275 , Creation speed 1.5668267949143413, Csi creation time is 4.556684
>03:09:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 293
>03:09:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:09:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 293
>03:09:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapo9zmt5or.yaml with timeout of 600
>03:09:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-293-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:09:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-293-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-293-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:09:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-293-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:09:20Z with timeout of 600
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:09:20Z with timeout of 600
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:09:20Z with timeout of 600
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:09:20Z with timeout of 600
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:09:20Z with timeout of 600
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:09:20Z with timeout of 600
>03:09:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 293 are : Creation time is 4.464584 , Creation speed 1.6126922463548674, Csi creation time is 4.426749
>03:09:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 294
>03:09:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:09:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 294
>03:09:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfvhhdd3g.yaml with timeout of 600
>03:09:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-294-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:09:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-294-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-294-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:09:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-294-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:09:45Z with timeout of 600
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:09:45Z with timeout of 600
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:09:45Z with timeout of 600
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:09:45Z with timeout of 600
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:09:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:09:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:09:45Z with timeout of 600
>03:09:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:09:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:09:45Z with timeout of 600
>03:09:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 294 are : Creation time is 4.577072 , Creation speed 1.5730580598251458, Csi creation time is 4.54003
>03:09:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 295
>03:09:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:10:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 295
>03:10:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapcqry1etg.yaml with timeout of 600
>03:10:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-295-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:10:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-295-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-295-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:10:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-295-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:10:09Z with timeout of 600
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:10:09Z with timeout of 600
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:10:09Z with timeout of 600
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:10:09Z with timeout of 600
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:10:09Z with timeout of 600
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:10:09Z with timeout of 600
>03:10:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 295 are : Creation time is 4.395256 , Creation speed 1.6381298381709735, Csi creation time is 4.36066
>03:10:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 296
>03:10:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:10:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 296
>03:10:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaph61rc099.yaml with timeout of 600
>03:10:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-296-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:10:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-296-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-296-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:10:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-296-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:10:34Z with timeout of 600
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:10:34Z with timeout of 600
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:10:34Z with timeout of 600
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:10:34Z with timeout of 600
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:10:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:10:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:10:34Z with timeout of 600
>03:10:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:10:34Z with timeout of 600
>03:10:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 296 are : Creation time is 4.986767 , Creation speed 1.44382121723353, Csi creation time is 4.943358
>03:10:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 297
>03:10:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:10:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 297
>03:10:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap97iit7a8.yaml with timeout of 600
>03:10:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:10:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-297-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:10:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-297-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-297-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:11:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-297-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:10:58Z with timeout of 600
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:10:58Z with timeout of 600
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:10:58Z with timeout of 600
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:10:58Z with timeout of 600
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:10:58Z with timeout of 600
>03:11:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:10:58Z with timeout of 600
>03:11:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 297 are : Creation time is 4.766841 , Creation speed 1.510434268732689, Csi creation time is 4.730602
>03:11:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 298
>03:11:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:11:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 298
>03:11:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsgb5syyd.yaml with timeout of 600
>03:11:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-298-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:11:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-298-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-298-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:11:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-298-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:11:23Z with timeout of 600
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:11:23Z with timeout of 600
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:11:23Z with timeout of 600
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:11:23Z with timeout of 600
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:11:23Z with timeout of 600
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:11:23Z with timeout of 600
>03:11:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 298 are : Creation time is 4.377602 , Creation speed 1.644736090672473, Csi creation time is 4.332479
>03:11:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 299
>03:11:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:11:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 299
>03:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapo950d3u7.yaml with timeout of 600
>03:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-299-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:11:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-299-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-299-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:11:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-299-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:11:48Z with timeout of 600
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:11:48Z with timeout of 600
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:11:48Z with timeout of 600
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:11:48Z with timeout of 600
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:11:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:11:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:11:48Z with timeout of 600
>03:11:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:11:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:11:48Z with timeout of 600
>03:11:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 299 are : Creation time is 4.542565 , Creation speed 1.5850075893245337, Csi creation time is 4.505129
>03:11:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 300
>03:11:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:12:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 300
>03:12:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapglpv9men.yaml with timeout of 600
>03:12:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-300-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:12:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-300-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-300-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:12:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-300-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:12:12Z with timeout of 600
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:12:12Z with timeout of 600
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:12:12Z with timeout of 600
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:12:12Z with timeout of 600
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:12:12Z with timeout of 600
>03:12:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:12:12Z with timeout of 600
>03:12:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 300 are : Creation time is 5.113784 , Creation speed 1.4079593506491475, Csi creation time is 5.070798
>03:12:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 301
>03:12:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:12:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 301
>03:12:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapqqwudatp.yaml with timeout of 600
>03:12:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-301-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:12:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-301-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-301-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:12:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-301-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:12:37Z with timeout of 600
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:12:37Z with timeout of 600
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:12:37Z with timeout of 600
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:12:37Z with timeout of 600
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:12:37Z with timeout of 600
>03:12:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:12:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:12:37Z with timeout of 600
>03:12:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 301 are : Creation time is 4.797764 , Creation speed 1.5006990756527416, Csi creation time is 4.755708
>03:12:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 302
>03:12:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:13:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 302
>03:13:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3tnix_1t.yaml with timeout of 600
>03:13:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-302-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:13:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-302-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-302-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:13:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-302-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:13:02Z with timeout of 600
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:13:02Z with timeout of 600
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:13:02Z with timeout of 600
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:13:02Z with timeout of 600
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:13:02Z with timeout of 600
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:13:02Z with timeout of 600
>03:13:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 302 are : Creation time is 4.634212 , Creation speed 1.553662197586127, Csi creation time is 4.593937
>03:13:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 303
>03:13:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:13:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 303
>03:13:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapzjhujj62.yaml with timeout of 600
>03:13:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-303-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:13:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-303-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-303-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:13:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-303-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:13:27Z with timeout of 600
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:13:27Z with timeout of 600
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:13:27Z with timeout of 600
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:13:27Z with timeout of 600
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:13:27Z with timeout of 600
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:13:27Z with timeout of 600
>03:13:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 303 are : Creation time is 5.369783 , Creation speed 1.340836305675667, Csi creation time is 5.331476
>03:13:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 304
>03:13:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:13:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 304
>03:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3a604mix.yaml with timeout of 600
>03:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:13:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-304-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:13:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-304-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-304-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:14:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-304-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:13:52Z with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:13:52Z with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:13:52Z with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:13:52Z with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:13:52Z with timeout of 600
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:13:52Z with timeout of 600
>03:14:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 304 are : Creation time is 5.261331 , Creation speed 1.3684750113611936, Csi creation time is 5.226548
>03:14:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 305
>03:14:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:14:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 305
>03:14:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapo1cs_lzo.yaml with timeout of 600
>03:14:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-305-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:14:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-305-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-305-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:14:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-305-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:14:17Z with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:14:17Z with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:14:17Z with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:14:17Z with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:14:17Z with timeout of 600
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:14:17Z with timeout of 600
>03:14:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 305 are : Creation time is 4.829668 , Creation speed 1.4907857020399746, Csi creation time is 4.789557
>03:14:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 306
>03:14:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:14:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 306
>03:14:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7awsjajh.yaml with timeout of 600
>03:14:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-306-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:14:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-306-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-306-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:14:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-306-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:14:42Z with timeout of 600
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:14:42Z with timeout of 600
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:14:42Z with timeout of 600
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:14:42Z with timeout of 600
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:14:42Z with timeout of 600
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:14:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:14:42Z with timeout of 600
>03:14:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 306 are : Creation time is 4.81457 , Creation speed 1.4954606538070898, Csi creation time is 4.773027
>03:14:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 307
>03:14:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:15:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 307
>03:15:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsj36tvkr.yaml with timeout of 600
>03:15:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-307-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:15:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-307-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-307-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:15:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-307-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:15:07Z with timeout of 600
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:15:07Z with timeout of 600
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:15:07Z with timeout of 600
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:15:07Z with timeout of 600
>03:15:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:15:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:15:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:15:07Z with timeout of 600
>03:15:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:15:07Z with timeout of 600
>03:15:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 307 are : Creation time is 5.557157 , Creation speed 1.2956265226985668, Csi creation time is 5.514741
>03:15:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 308
>03:15:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:15:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 308
>03:15:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5rct95i1.yaml with timeout of 600
>03:15:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-308-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:15:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-308-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-308-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:15:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-308-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:15:31Z with timeout of 600
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:15:31Z with timeout of 600
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:15:31Z with timeout of 600
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:15:31Z with timeout of 600
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:15:31Z with timeout of 600
>03:15:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:15:31Z with timeout of 600
>03:15:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 308 are : Creation time is 5.872486 , Creation speed 1.2260565627572377, Csi creation time is 5.835109
>03:15:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 309
>03:15:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:15:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 309
>03:15:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvgviz7hj.yaml with timeout of 600
>03:15:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:15:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-309-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:15:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-309-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:16:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-309-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:16:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-309-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:15:56Z with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:15:56Z with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:15:56Z with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:15:56Z with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:15:56Z with timeout of 600
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:15:56Z with timeout of 600
>03:16:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 309 are : Creation time is 5.93821 , Creation speed 1.2124865910771092, Csi creation time is 5.905036
>03:16:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 310
>03:16:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:16:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 310
>03:16:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapuqigxmv2.yaml with timeout of 600
>03:16:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-310-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:16:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-310-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-310-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:16:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-310-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:16:21Z with timeout of 600
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:16:21Z with timeout of 600
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:16:21Z with timeout of 600
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:16:21Z with timeout of 600
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:16:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:16:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:16:21Z with timeout of 600
>03:16:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:16:21Z with timeout of 600
>03:16:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 310 are : Creation time is 6.02534 , Creation speed 1.1949533138378914, Csi creation time is 5.983072
>03:16:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 311
>03:16:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:16:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 311
>03:16:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwu7lr0i5.yaml with timeout of 600
>03:16:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-311-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:16:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-311-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:16:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-311-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:16:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-311-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:16:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:16:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:16:45Z with timeout of 600
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:16:45Z with timeout of 600
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:16:45Z with timeout of 600
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:16:45Z with timeout of 600
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:16:45Z with timeout of 600
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:16:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:16:45Z with timeout of 600
>03:16:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 311 are : Creation time is 6.106368 , Creation speed 1.1790969689347253, Csi creation time is 6.067688
>03:16:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 312
>03:16:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:17:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 312
>03:17:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapk_c9mrvs.yaml with timeout of 600
>03:17:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-312-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:17:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-312-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-312-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:17:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-312-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:17:12Z with timeout of 600
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:17:12Z with timeout of 600
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:17:12Z with timeout of 600
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:17:12Z with timeout of 600
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:17:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:17:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:17:12Z with timeout of 600
>03:17:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:17:12Z with timeout of 600
>03:17:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 312 are : Creation time is 6.143302 , Creation speed 1.1720081480610915, Csi creation time is 6.107647
>03:17:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 313
>03:17:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:17:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 313
>03:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapt53avpv6.yaml with timeout of 600
>03:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-313-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:17:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-313-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-313-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:17:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-313-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:17:39Z with timeout of 600
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:17:39Z with timeout of 600
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:17:39Z with timeout of 600
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:17:39Z with timeout of 600
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:17:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:17:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:17:39Z with timeout of 600
>03:17:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:17:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:17:39Z with timeout of 600
>03:17:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 313 are : Creation time is 5.751081 , Creation speed 1.2519385485963421, Csi creation time is 5.710127
>03:17:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 314
>03:17:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:18:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 314
>03:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwh_shp__.yaml with timeout of 600
>03:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-314-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:18:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-314-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:18:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-314-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:18:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-314-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:18:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:18:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:18:03Z with timeout of 600
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:18:03Z with timeout of 600
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:18:03Z with timeout of 600
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:18:03Z with timeout of 600
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:18:03Z with timeout of 600
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:18:03Z with timeout of 600
>03:18:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 314 are : Creation time is 6.135975 , Creation speed 1.173407649151113, Csi creation time is 6.098018
>03:18:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 315
>03:18:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:18:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 315
>03:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapx99g7pbn.yaml with timeout of 600
>03:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-315-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:18:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-315-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:18:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-315-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:18:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-315-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:18:28Z with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:18:28Z with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:18:28Z with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:18:28Z with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:18:28Z with timeout of 600
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:18:28Z with timeout of 600
>03:18:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 315 are : Creation time is 6.003205 , Creation speed 1.1993593422180318, Csi creation time is 5.966816
>03:18:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 316
>03:18:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:18:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 316
>03:18:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapqqqug6mi.yaml with timeout of 600
>03:18:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:18:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-316-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:18:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-316-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-316-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:19:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-316-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:18:55Z with timeout of 600
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:18:55Z with timeout of 600
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:18:55Z with timeout of 600
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:18:55Z with timeout of 600
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:18:55Z with timeout of 600
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:18:55Z with timeout of 600
>03:19:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 316 are : Creation time is 6.456799 , Creation speed 1.1151036295229262, Csi creation time is 6.416137
>03:19:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 317
>03:19:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:19:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 317
>03:19:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0mybzpmt.yaml with timeout of 600
>03:19:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-317-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:19:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-317-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-317-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:19:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-317-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:19:21Z with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:19:21Z with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:19:21Z with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:19:21Z with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:19:21Z with timeout of 600
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:19:21Z with timeout of 600
>03:19:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 317 are : Creation time is 6.412037 , Creation speed 1.1228880931285956, Csi creation time is 6.376411
>03:19:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 318
>03:19:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:19:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 318
>03:19:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7q36g8me.yaml with timeout of 600
>03:19:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-318-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:19:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-318-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-318-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:19:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-318-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:19:47Z with timeout of 600
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:19:47Z with timeout of 600
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:19:47Z with timeout of 600
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:19:47Z with timeout of 600
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:19:47Z with timeout of 600
>03:19:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:19:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:19:47Z with timeout of 600
>03:19:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 318 are : Creation time is 6.842811 , Creation speed 1.0521991620110507, Csi creation time is 6.804608
>03:19:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 319
>03:19:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:20:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 319
>03:20:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapl_p99fx_.yaml with timeout of 600
>03:20:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-319-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:20:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-319-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-319-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:20:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-319-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:20:12Z with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:20:12Z with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:20:12Z with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:20:12Z with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:20:12Z with timeout of 600
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:20:12Z with timeout of 600
>03:20:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 319 are : Creation time is 5.892548 , Creation speed 1.2218822825032567, Csi creation time is 5.849648
>03:20:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 320
>03:20:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:20:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 320
>03:20:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snape4f5cml9.yaml with timeout of 600
>03:20:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-320-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:20:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-320-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:20:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-320-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:20:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-320-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:20:37Z with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:20:37Z with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:20:37Z with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:20:37Z with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:20:37Z with timeout of 600
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:20:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:20:37Z with timeout of 600
>03:20:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 320 are : Creation time is 5.995439 , Creation speed 1.200912893951552, Csi creation time is 5.952921
>03:20:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 321
>03:20:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:21:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 321
>03:21:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfkv26v4t.yaml with timeout of 600
>03:21:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-321-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:21:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-321-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-321-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:21:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-321-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:21:02Z with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:21:02Z with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:21:02Z with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:21:02Z with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:21:02Z with timeout of 600
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:21:02Z with timeout of 600
>03:21:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 321 are : Creation time is 6.897744 , Creation speed 1.0438195444771508, Csi creation time is 6.864348
>03:21:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 322
>03:21:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:21:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 322
>03:21:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvbgi5n_5.yaml with timeout of 600
>03:21:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-322-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:21:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-322-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-322-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:21:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-322-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:21:27Z with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:21:27Z with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:21:27Z with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:21:27Z with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:21:27Z with timeout of 600
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:21:27Z with timeout of 600
>03:21:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 322 are : Creation time is 7.098734 , Creation speed 1.01426536055584, Csi creation time is 7.055001
>03:21:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 323
>03:21:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:21:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 323
>03:21:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapzj_sye4b.yaml with timeout of 600
>03:21:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:21:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-323-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:21:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-323-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-323-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:22:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-323-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:21:52Z with timeout of 600
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:21:52Z with timeout of 600
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:21:52Z with timeout of 600
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:21:52Z with timeout of 600
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:22:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:22:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:21:52Z with timeout of 600
>03:22:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:21:52Z with timeout of 600
>03:22:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 323 are : Creation time is 6.679088 , Creation speed 1.077991486262795, Csi creation time is 6.641206
>03:22:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 324
>03:22:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:22:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 324
>03:22:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapyvi6dbpw.yaml with timeout of 600
>03:22:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-324-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:22:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-324-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-324-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:22:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-324-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:22:16Z with timeout of 600
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:22:16Z with timeout of 600
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:22:16Z with timeout of 600
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:22:16Z with timeout of 600
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:22:16Z with timeout of 600
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:22:16Z with timeout of 600
>03:22:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 324 are : Creation time is 6.513681 , Creation speed 1.105365767835422, Csi creation time is 6.471035
>03:22:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 325
>03:22:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:22:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 325
>03:22:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4yi0nvw1.yaml with timeout of 600
>03:22:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-325-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:22:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-325-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-325-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:22:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-325-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:22:40Z with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:22:40Z with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:22:40Z with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:22:40Z with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:22:40Z with timeout of 600
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:22:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:22:40Z with timeout of 600
>03:22:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 325 are : Creation time is 6.62841 , Creation speed 1.0862333500794308, Csi creation time is 6.590888
>03:22:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 326
>03:22:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:23:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 326
>03:23:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapyorjogjs.yaml with timeout of 600
>03:23:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-326-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:23:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-326-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-326-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:23:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-326-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:23:05Z with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:23:05Z with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:23:05Z with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:23:05Z with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:23:05Z with timeout of 600
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:23:05Z with timeout of 600
>03:23:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 326 are : Creation time is 6.949335 , Creation speed 1.0360703578112151, Csi creation time is 6.914569
>03:23:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 327
>03:23:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:23:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 327
>03:23:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapavv3hpkm.yaml with timeout of 600
>03:23:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-327-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:23:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-327-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:23:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-327-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:23:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-327-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:23:30Z with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:23:30Z with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:23:30Z with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:23:30Z with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:23:30Z with timeout of 600
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:23:30Z with timeout of 600
>03:23:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 327 are : Creation time is 7.029893 , Creation speed 1.0241976655974707, Csi creation time is 6.955849
>03:23:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 328
>03:23:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:23:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 328
>03:23:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapz_e60brz.yaml with timeout of 600
>03:23:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:23:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-328-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:23:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-328-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-328-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:24:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-328-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:23:55Z with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:23:55Z with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:23:55Z with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:23:55Z with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:23:55Z with timeout of 600
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:23:55Z with timeout of 600
>03:24:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 328 are : Creation time is 7.00653 , Creation speed 1.0276128126190853, Csi creation time is 6.955662
>03:24:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 329
>03:24:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:24:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 329
>03:24:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapx2ykv9kv.yaml with timeout of 600
>03:24:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-329-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:24:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-329-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:24:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-329-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:24:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-329-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:24:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:24:20Z with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:24:20Z with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:24:20Z with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:24:20Z with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:24:20Z with timeout of 600
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:24:20Z with timeout of 600
>03:24:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 329 are : Creation time is 7.203355 , Creation speed 0.9995342448067602, Csi creation time is 7.155177
>03:24:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 330
>03:24:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:24:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 330
>03:24:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapntw10n_0.yaml with timeout of 600
>03:24:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-330-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:24:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-330-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-330-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:24:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-330-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:24:45Z with timeout of 600
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:24:45Z with timeout of 600
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:24:45Z with timeout of 600
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:24:45Z with timeout of 600
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:24:45Z with timeout of 600
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:24:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:24:45Z with timeout of 600
>03:24:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 330 are : Creation time is 7.500632 , Creation speed 0.9599191108162618, Csi creation time is 7.466375
>03:24:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 331
>03:24:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:25:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 331
>03:25:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapra4f0i0r.yaml with timeout of 600
>03:25:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-331-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:25:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-331-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-331-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:25:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-331-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:25:10Z with timeout of 600
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:25:10Z with timeout of 600
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:25:10Z with timeout of 600
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:25:10Z with timeout of 600
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:25:10Z with timeout of 600
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:25:10Z with timeout of 600
>03:25:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 331 are : Creation time is 6.749593 , Creation speed 1.0667309865943029, Csi creation time is 6.708602
>03:25:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 332
>03:25:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:25:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 332
>03:25:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapb_k2o85m.yaml with timeout of 600
>03:25:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-332-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:25:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-332-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:25:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-332-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:25:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-332-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:25:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:25:36Z with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:25:36Z with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:25:36Z with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:25:36Z with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:25:36Z with timeout of 600
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:25:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:25:36Z with timeout of 600
>03:25:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 332 are : Creation time is 7.400005 , Creation speed 0.9729723155592462, Csi creation time is 7.357969
>03:25:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 333
>03:25:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:26:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 333
>03:26:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapv_xu7fra.yaml with timeout of 600
>03:26:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-333-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:26:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-333-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-333-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:26:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-333-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:26:02Z with timeout of 600
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:26:02Z with timeout of 600
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:26:02Z with timeout of 600
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:26:02Z with timeout of 600
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:26:02Z with timeout of 600
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:26:02Z with timeout of 600
>03:26:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 333 are : Creation time is 7.528707 , Creation speed 0.9563395148728726, Csi creation time is 7.491237
>03:26:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 334
>03:26:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:26:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 334
>03:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapkp9rbhs9.yaml with timeout of 600
>03:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-334-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:26:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-334-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-334-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:26:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-334-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:26:27Z with timeout of 600
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:26:27Z with timeout of 600
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:26:27Z with timeout of 600
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:26:27Z with timeout of 600
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:26:27Z with timeout of 600
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:26:27Z with timeout of 600
>03:26:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 334 are : Creation time is 7.676838 , Creation speed 0.9378861453113899, Csi creation time is 7.637678
>03:26:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 335
>03:26:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:26:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 335
>03:26:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8_ditkq0.yaml with timeout of 600
>03:26:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:26:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-335-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:26:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-335-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-335-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:27:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-335-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:26:52Z with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:26:52Z with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:26:52Z with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:26:52Z with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:26:52Z with timeout of 600
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:26:52Z with timeout of 600
>03:27:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 335 are : Creation time is 7.412583 , Creation speed 0.9713213329280765, Csi creation time is 7.374841
>03:27:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 336
>03:27:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:27:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 336
>03:27:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1bdpugx5.yaml with timeout of 600
>03:27:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-336-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:27:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-336-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-336-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:27:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-336-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:27:17Z with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:27:17Z with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:27:17Z with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:27:17Z with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:27:17Z with timeout of 600
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:27:17Z with timeout of 600
>03:27:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 336 are : Creation time is 7.411992 , Creation speed 0.9713987818659277, Csi creation time is 7.372543
>03:27:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 337
>03:27:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:27:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 337
>03:27:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapxua6_ee4.yaml with timeout of 600
>03:27:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-337-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:27:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-337-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-337-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:27:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-337-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:27:42Z with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:27:42Z with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:27:42Z with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:27:42Z with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:27:42Z with timeout of 600
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:27:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:27:42Z with timeout of 600
>03:27:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 337 are : Creation time is 7.989621 , Creation speed 0.901169154331601, Csi creation time is 7.951449
>03:27:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 338
>03:27:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:28:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 338
>03:28:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_bhzg0g3.yaml with timeout of 600
>03:28:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-338-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:28:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-338-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-338-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:28:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-338-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:28:08Z with timeout of 600
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:28:08Z with timeout of 600
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:28:08Z with timeout of 600
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:28:08Z with timeout of 600
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:28:08Z with timeout of 600
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:28:08Z with timeout of 600
>03:28:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 338 are : Creation time is 7.384094 , Creation speed 0.9750688439231678, Csi creation time is 7.343229
>03:28:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 339
>03:28:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:28:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 339
>03:28:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap44k1cj1j.yaml with timeout of 600
>03:28:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-339-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:28:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-339-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-339-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:28:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-339-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:28:33Z with timeout of 600
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:28:33Z with timeout of 600
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:28:33Z with timeout of 600
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:28:33Z with timeout of 600
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:28:33Z with timeout of 600
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:28:33Z with timeout of 600
>03:28:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 339 are : Creation time is 7.550386 , Creation speed 0.9535936308421848, Csi creation time is 7.509788
>03:28:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 340
>03:28:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:28:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 340
>03:28:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbz6kepux.yaml with timeout of 600
>03:28:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:28:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-340-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:28:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-340-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-340-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:29:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-340-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:28:58Z with timeout of 600
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:28:58Z with timeout of 600
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:28:58Z with timeout of 600
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:28:58Z with timeout of 600
>03:29:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:29:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:29:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:28:58Z with timeout of 600
>03:29:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:28:58Z with timeout of 600
>03:29:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 340 are : Creation time is 7.761406 , Creation speed 0.9276669716801311, Csi creation time is 7.721356
>03:29:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 341
>03:29:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:29:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 341
>03:29:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmttzw_8x.yaml with timeout of 600
>03:29:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-341-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:29:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-341-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-341-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:29:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-341-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:29:22Z with timeout of 600
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:29:22Z with timeout of 600
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:29:22Z with timeout of 600
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:29:22Z with timeout of 600
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:29:22Z with timeout of 600
>03:29:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:29:22Z with timeout of 600
>03:29:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 341 are : Creation time is 8.125037 , Creation speed 0.8861498107639386, Csi creation time is 8.083623
>03:29:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 342
>03:29:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:29:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 342
>03:29:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapy8wwbadr.yaml with timeout of 600
>03:29:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-342-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:29:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-342-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-342-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:29:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-342-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:29:47Z with timeout of 600
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:29:47Z with timeout of 600
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:29:47Z with timeout of 600
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:29:47Z with timeout of 600
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:29:47Z with timeout of 600
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:29:47Z with timeout of 600
>03:29:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 342 are : Creation time is 7.848938 , Creation speed 0.9173215535655906, Csi creation time is 7.811559
>03:29:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 343
>03:29:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:30:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 343
>03:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snape6u_obv_.yaml with timeout of 600
>03:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-343-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:30:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-343-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-343-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:30:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-343-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:30:15Z with timeout of 600
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:30:15Z with timeout of 600
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:30:15Z with timeout of 600
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:30:15Z with timeout of 600
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:30:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:30:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:30:15Z with timeout of 600
>03:30:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:30:15Z with timeout of 600
>03:30:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 343 are : Creation time is 8.102849 , Creation speed 0.888576351354937, Csi creation time is 8.054154
>03:30:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 344
>03:30:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:30:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 344
>03:30:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8lhwkztq.yaml with timeout of 600
>03:30:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-344-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:30:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-344-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-344-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:30:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-344-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:30:41Z with timeout of 600
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:30:41Z with timeout of 600
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:30:41Z with timeout of 600
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:30:41Z with timeout of 600
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:30:41Z with timeout of 600
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:30:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:30:41Z with timeout of 600
>03:30:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 344 are : Creation time is 8.381392 , Creation speed 0.8590458482314155, Csi creation time is 8.343683
>03:30:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 345
>03:30:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:31:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 345
>03:31:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap066kajt2.yaml with timeout of 600
>03:31:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-345-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:31:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-345-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:31:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-345-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:31:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-345-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:31:06Z with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:31:06Z with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:31:06Z with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:31:06Z with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:31:06Z with timeout of 600
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:31:06Z with timeout of 600
>03:31:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 345 are : Creation time is 8.452568 , Creation speed 0.8518121356728512, Csi creation time is 8.412372
>03:31:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 346
>03:31:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:31:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 346
>03:31:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap59lvtork.yaml with timeout of 600
>03:31:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-346-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:31:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-346-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-346-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:31:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-346-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:31:32Z with timeout of 600
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:31:32Z with timeout of 600
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:31:32Z with timeout of 600
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:31:32Z with timeout of 600
>03:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:31:32Z with timeout of 600
>03:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:31:32Z with timeout of 600
>03:31:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 346 are : Creation time is 8.086707 , Creation speed 0.8903500522524187, Csi creation time is 8.049001
>03:31:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 347
>03:31:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:31:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 347
>03:31:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapn0x8s4vz.yaml with timeout of 600
>03:31:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:31:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-347-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:31:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-347-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-347-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:32:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-347-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:31:56Z with timeout of 600
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:31:56Z with timeout of 600
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:31:56Z with timeout of 600
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:31:56Z with timeout of 600
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:31:56Z with timeout of 600
>03:32:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:31:56Z with timeout of 600
>03:32:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 347 are : Creation time is 7.952928 , Creation speed 0.9053269437369482, Csi creation time is 7.913782
>03:32:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 348
>03:32:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:32:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 348
>03:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphi5onu7r.yaml with timeout of 600
>03:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-348-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:32:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-348-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-348-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:32:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-348-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:32:21Z with timeout of 600
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:32:21Z with timeout of 600
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:32:21Z with timeout of 600
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:32:21Z with timeout of 600
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:32:21Z with timeout of 600
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:32:21Z with timeout of 600
>03:32:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 348 are : Creation time is 8.557894 , Creation speed 0.8413284857232399, Csi creation time is 8.514457
>03:32:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 349
>03:32:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:32:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 349
>03:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapgyv6e9y8.yaml with timeout of 600
>03:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-349-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:32:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-349-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-349-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:32:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-349-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:32:46Z with timeout of 600
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:32:46Z with timeout of 600
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:32:46Z with timeout of 600
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:32:46Z with timeout of 600
>03:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:32:46Z with timeout of 600
>03:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:32:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:32:46Z with timeout of 600
>03:32:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 349 are : Creation time is 8.814625 , Creation speed 0.8168243118680603, Csi creation time is 8.770596
>03:32:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 350
>03:32:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:33:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 350
>03:33:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapw59hbv35.yaml with timeout of 600
>03:33:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-350-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:33:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-350-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-350-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:33:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-350-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:33:10Z with timeout of 600
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:33:10Z with timeout of 600
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:33:10Z with timeout of 600
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:33:10Z with timeout of 600
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:33:10Z with timeout of 600
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:33:10Z with timeout of 600
>03:33:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 350 are : Creation time is 8.561534 , Creation speed 0.8409707886460535, Csi creation time is 8.520554
>03:33:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 351
>03:33:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:33:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 351
>03:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaphm045gvq.yaml with timeout of 600
>03:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-351-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:33:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-351-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-351-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:33:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-351-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:33:35Z with timeout of 600
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:33:35Z with timeout of 600
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:33:35Z with timeout of 600
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:33:35Z with timeout of 600
>03:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:33:35Z with timeout of 600
>03:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:33:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:33:35Z with timeout of 600
>03:33:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 351 are : Creation time is 8.928436 , Creation speed 0.8064122316607298, Csi creation time is 8.892146
>03:33:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 352
>03:33:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:34:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 352
>03:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapgdsicjxy.yaml with timeout of 600
>03:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-352-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:34:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-352-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-352-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:34:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-352-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:34:00Z with timeout of 600
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:34:00Z with timeout of 600
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:34:00Z with timeout of 600
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:34:00Z with timeout of 600
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:34:00Z with timeout of 600
>03:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:34:00Z with timeout of 600
>03:34:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 352 are : Creation time is 9.040525 , Creation speed 0.7964139250762539, Csi creation time is 8.99427
>03:34:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 353
>03:34:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:34:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 353
>03:34:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapi9tlw3w3.yaml with timeout of 600
>03:34:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-353-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:34:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-353-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-353-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:34:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-353-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:34:25Z with timeout of 600
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:34:25Z with timeout of 600
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:34:25Z with timeout of 600
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:34:25Z with timeout of 600
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:34:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:34:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:34:25Z with timeout of 600
>03:34:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:34:25Z with timeout of 600
>03:34:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 353 are : Creation time is 9.152239 , Creation speed 0.7866927426174076, Csi creation time is 9.11353
>03:34:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 354
>03:34:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:34:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 354
>03:34:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapzjgs93q9.yaml with timeout of 600
>03:34:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:34:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-354-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:34:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-354-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-354-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:35:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-354-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:34:49Z with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:34:49Z with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:34:49Z with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:34:49Z with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:34:49Z with timeout of 600
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:34:49Z with timeout of 600
>03:35:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 354 are : Creation time is 9.426124 , Creation speed 0.7638346365908193, Csi creation time is 9.388555
>03:35:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 355
>03:35:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:35:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 355
>03:35:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfyzgce7o.yaml with timeout of 600
>03:35:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-355-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:35:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-355-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:35:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-355-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:35:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-355-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:35:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:35:14Z with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:35:14Z with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:35:14Z with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:35:14Z with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:35:14Z with timeout of 600
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:35:14Z with timeout of 600
>03:35:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 355 are : Creation time is 8.664134 , Creation speed 0.8310120780680446, Csi creation time is 8.624757
>03:35:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 356
>03:35:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:35:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 356
>03:35:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7aqx8lp8.yaml with timeout of 600
>03:35:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-356-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:35:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-356-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:35:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-356-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:35:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-356-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:35:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:35:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:35:39Z with timeout of 600
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:35:39Z with timeout of 600
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:35:39Z with timeout of 600
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:35:39Z with timeout of 600
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:35:39Z with timeout of 600
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:35:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:35:39Z with timeout of 600
>03:35:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 356 are : Creation time is 9.295734 , Creation speed 0.7745488414362977, Csi creation time is 9.257891
>03:35:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 357
>03:35:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:36:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 357
>03:36:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap6lzxxpwz.yaml with timeout of 600
>03:36:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-357-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:36:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-357-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:36:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-357-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:36:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-357-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:36:04Z with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:36:04Z with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:36:04Z with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:36:04Z with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:36:04Z with timeout of 600
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:36:04Z with timeout of 600
>03:36:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 357 are : Creation time is 9.214386 , Creation speed 0.7813868444408559, Csi creation time is 9.176344
>03:36:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 358
>03:36:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:36:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 358
>03:36:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapg3nkn5eg.yaml with timeout of 600
>03:36:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-358-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:36:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-358-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:36:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-358-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:36:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-358-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:36:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:36:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:36:29Z with timeout of 600
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:36:29Z with timeout of 600
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:36:29Z with timeout of 600
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:36:29Z with timeout of 600
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:36:29Z with timeout of 600
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:36:29Z with timeout of 600
>03:36:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 358 are : Creation time is 9.421861 , Creation speed 0.7641802399759453, Csi creation time is 9.379907
>03:36:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 359
>03:36:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:36:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 359
>03:36:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapd6974fr5.yaml with timeout of 600
>03:36:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:36:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-359-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:36:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-359-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-359-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:37:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-359-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:36:54Z with timeout of 600
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:36:54Z with timeout of 600
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:36:54Z with timeout of 600
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:36:54Z with timeout of 600
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:37:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:37:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:36:54Z with timeout of 600
>03:37:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:36:54Z with timeout of 600
>03:37:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 359 are : Creation time is 8.927768 , Creation speed 0.8064725696277054, Csi creation time is 8.892152
>03:37:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 360
>03:37:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:37:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 360
>03:37:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8neoi5zv.yaml with timeout of 600
>03:37:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-360-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:37:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-360-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:37:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-360-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:37:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-360-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:37:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:37:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:37:18Z with timeout of 600
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:37:18Z with timeout of 600
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:37:18Z with timeout of 600
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:37:18Z with timeout of 600
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:37:18Z with timeout of 600
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:37:18Z with timeout of 600
>03:37:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 360 are : Creation time is 9.543833 , Creation speed 0.754413871240203, Csi creation time is 9.50614
>03:37:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 361
>03:37:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:37:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 361
>03:37:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7jcr9bjl.yaml with timeout of 600
>03:37:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-361-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:37:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-361-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-361-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:37:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-361-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:37:43Z with timeout of 600
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:37:43Z with timeout of 600
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:37:43Z with timeout of 600
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:37:43Z with timeout of 600
>03:37:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:37:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:37:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:37:43Z with timeout of 600
>03:37:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:37:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:37:43Z with timeout of 600
>03:37:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 361 are : Creation time is 10.006439 , Creation speed 0.7195366903250997, Csi creation time is 9.963555
>03:37:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 362
>03:37:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:38:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 362
>03:38:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap74vfo7d5.yaml with timeout of 600
>03:38:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-362-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:38:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-362-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-362-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:38:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-362-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:38:07Z with timeout of 600
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:38:07Z with timeout of 600
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:38:07Z with timeout of 600
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:38:07Z with timeout of 600
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:38:07Z with timeout of 600
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:38:07Z with timeout of 600
>03:38:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 362 are : Creation time is 9.738376 , Creation speed 0.7393429869620971, Csi creation time is 9.703789
>03:38:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 363
>03:38:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:38:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 363
>03:38:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap6yxs4uye.yaml with timeout of 600
>03:38:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-363-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:38:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-363-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:38:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-363-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:38:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-363-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:38:31Z with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:38:31Z with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:38:31Z with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:38:31Z with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:38:31Z with timeout of 600
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:38:31Z with timeout of 600
>03:38:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 363 are : Creation time is 9.452459 , Creation speed 0.7617065569922071, Csi creation time is 9.409512
>03:38:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 364
>03:38:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:38:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 364
>03:38:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaplkkl9ae6.yaml with timeout of 600
>03:38:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:38:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:38:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:39:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:39:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:39:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:38:55Z with timeout of 600
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:38:55Z with timeout of 600
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:38:55Z with timeout of 600
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:38:55Z with timeout of 600
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:39:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:39:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:38:55Z with timeout of 600
>03:39:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:38:55Z with timeout of 600
>03:39:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 364 are : Creation time is 10.33436 , Creation speed 0.6967049725382123, Csi creation time is 10.286143
>03:39:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 365
>03:39:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:39:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 365
>03:39:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapooel0se2.yaml with timeout of 600
>03:39:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-365-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:39:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-365-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-365-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:39:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-365-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:39:30Z with timeout of 600
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:39:30Z with timeout of 600
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:39:30Z with timeout of 600
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:39:30Z with timeout of 600
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:39:30Z with timeout of 600
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:39:30Z with timeout of 600
>03:39:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 365 are : Creation time is 9.869283 , Creation speed 0.7295362793832136, Csi creation time is 9.830578
>03:39:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 366
>03:39:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:39:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 366
>03:39:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapoivjyv3b.yaml with timeout of 600
>03:39:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:39:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-366-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:39:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-366-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-366-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:40:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-366-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:39:54Z with timeout of 600
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:39:54Z with timeout of 600
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:39:54Z with timeout of 600
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:39:54Z with timeout of 600
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:39:54Z with timeout of 600
>03:40:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:39:54Z with timeout of 600
>03:40:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 366 are : Creation time is 10.06501 , Creation speed 0.7153495128171756, Csi creation time is 10.021028
>03:40:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 367
>03:40:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:40:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 367
>03:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprw_o87t3.yaml with timeout of 600
>03:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:40:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:40:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:40:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:40:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:40:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:40:19Z with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:40:19Z with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:40:19Z with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:40:19Z with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:40:19Z with timeout of 600
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:40:19Z with timeout of 600
>03:40:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 367 are : Creation time is 10.379508 , Creation speed 0.6936744978663729, Csi creation time is 10.339466
>03:40:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 368
>03:40:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:40:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 368
>03:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapw57r_h33.yaml with timeout of 600
>03:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:40:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:40:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:41:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:41:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:41:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:40:57Z with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:40:57Z with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:40:57Z with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:40:57Z with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:40:57Z with timeout of 600
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:40:57Z with timeout of 600
>03:41:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 368 are : Creation time is 10.40552 , Creation speed 0.6919404316170649, Csi creation time is 10.362107
>03:41:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 369
>03:41:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:41:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 369
>03:41:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_qbqzvr7.yaml with timeout of 600
>03:41:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:41:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:41:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:41:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:41:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:41:33Z with timeout of 600
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:41:33Z with timeout of 600
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:41:33Z with timeout of 600
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:41:33Z with timeout of 600
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:41:33Z with timeout of 600
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:41:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:41:33Z with timeout of 600
>03:41:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 369 are : Creation time is 10.829412 , Creation speed 0.6648560420454961, Csi creation time is 10.789777
>03:41:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 370
>03:41:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:42:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 370
>03:42:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap76vfngv_.yaml with timeout of 600
>03:42:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:42:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:42:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:42:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:42:07Z with timeout of 600
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:42:07Z with timeout of 600
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:42:07Z with timeout of 600
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:42:07Z with timeout of 600
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:42:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:42:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:42:07Z with timeout of 600
>03:42:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:42:07Z with timeout of 600
>03:42:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 370 are : Creation time is 10.305108 , Creation speed 0.6986826338937933, Csi creation time is 10.264255
>03:42:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 371
>03:42:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:42:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 371
>03:42:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapd2e0xd6n.yaml with timeout of 600
>03:42:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:42:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:42:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:42:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:42:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:43:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:43:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:43:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:42:43Z with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:42:43Z with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:42:43Z with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:42:43Z with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:42:43Z with timeout of 600
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:42:43Z with timeout of 600
>03:43:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 371 are : Creation time is 10.690471 , Creation speed 0.6734969862412984, Csi creation time is 10.647072
>03:43:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 372
>03:43:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:43:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 372
>03:43:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapq8m8bmmr.yaml with timeout of 600
>03:43:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:43:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:43:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:43:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:43:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:43:17Z with timeout of 600
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:43:17Z with timeout of 600
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:43:17Z with timeout of 600
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:43:17Z with timeout of 600
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:43:17Z with timeout of 600
>03:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:43:17Z with timeout of 600
>03:43:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 372 are : Creation time is 10.60265 , Creation speed 0.6790755141403328, Csi creation time is 10.565591
>03:43:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 373
>03:43:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:43:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 373
>03:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0vcmt4ka.yaml with timeout of 600
>03:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:43:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:43:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:44:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:44:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:44:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:44:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:44:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:44:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:43:52Z with timeout of 600
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:43:52Z with timeout of 600
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:43:52Z with timeout of 600
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:43:52Z with timeout of 600
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:43:52Z with timeout of 600
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:43:52Z with timeout of 600
>03:44:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 373 are : Creation time is 10.570162 , Creation speed 0.6811626917354721, Csi creation time is 10.525101
>03:44:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 374
>03:44:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:44:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 374
>03:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapa__3yhbq.yaml with timeout of 600
>03:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:44:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:44:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:44:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:44:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:44:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:44:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:44:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:44:27Z with timeout of 600
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:44:27Z with timeout of 600
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:44:27Z with timeout of 600
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:44:27Z with timeout of 600
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:44:27Z with timeout of 600
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:44:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:44:27Z with timeout of 600
>03:44:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 374 are : Creation time is 10.631161 , Creation speed 0.6772543469146972, Csi creation time is 10.588943
>03:44:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 375
>03:44:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:45:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 375
>03:45:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_ws27npx.yaml with timeout of 600
>03:45:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:45:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:45:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:45:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:45:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:45:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:45:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:45:02Z with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:45:02Z with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:45:02Z with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:45:02Z with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:45:02Z with timeout of 600
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:45:02Z with timeout of 600
>03:45:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 375 are : Creation time is 10.583489 , Creation speed 0.6803049542546886, Csi creation time is 10.546633
>03:45:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 376
>03:45:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:45:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 376
>03:45:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_eisopfz.yaml with timeout of 600
>03:45:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:45:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:45:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:45:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:45:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:45:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:45:37Z with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:45:37Z with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:45:37Z with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:45:37Z with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:45:37Z with timeout of 600
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:45:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:45:37Z with timeout of 600
>03:45:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 376 are : Creation time is 11.52329 , Creation speed 0.624821557037964, Csi creation time is 11.485574
>03:45:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 377
>03:45:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:46:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 377
>03:46:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdzqnj9y7.yaml with timeout of 600
>03:46:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:46:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:46:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:46:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:46:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:46:11Z with timeout of 600
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:46:11Z with timeout of 600
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:46:11Z with timeout of 600
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:46:11Z with timeout of 600
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:46:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:46:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:46:11Z with timeout of 600
>03:46:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:46:11Z with timeout of 600
>03:46:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 377 are : Creation time is 11.383365 , Creation speed 0.6325019007999831, Csi creation time is 11.343073
>03:46:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 378
>03:46:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:46:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 378
>03:46:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprpacvmbd.yaml with timeout of 600
>03:46:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:46:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:46:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:46:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:46:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:47:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:46:46Z with timeout of 600
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:46:46Z with timeout of 600
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:46:46Z with timeout of 600
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:46:46Z with timeout of 600
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:46:46Z with timeout of 600
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:46:46Z with timeout of 600
>03:47:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 378 are : Creation time is 11.016081 , Creation speed 0.6535899654332608, Csi creation time is 10.970923
>03:47:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 379
>03:47:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:47:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 379
>03:47:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapfv5fofg0.yaml with timeout of 600
>03:47:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:47:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:47:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:47:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:47:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:47:21Z with timeout of 600
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:47:21Z with timeout of 600
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:47:21Z with timeout of 600
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:47:21Z with timeout of 600
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:47:21Z with timeout of 600
>03:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:47:21Z with timeout of 600
>03:47:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 379 are : Creation time is 11.395765 , Creation speed 0.6318136606011092, Csi creation time is 11.352032
>03:47:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 380
>03:47:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:47:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 380
>03:47:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmzftk0re.yaml with timeout of 600
>03:47:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:47:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:47:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:48:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:48:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:48:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:47:55Z with timeout of 600
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:47:55Z with timeout of 600
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:47:55Z with timeout of 600
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:47:55Z with timeout of 600
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:47:55Z with timeout of 600
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:47:55Z with timeout of 600
>03:48:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 380 are : Creation time is 11.667397 , Creation speed 0.6171042264182834, Csi creation time is 11.624941
>03:48:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 381
>03:48:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:48:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 381
>03:48:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap22blgajr.yaml with timeout of 600
>03:48:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:48:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:48:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:48:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:48:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:48:30Z with timeout of 600
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:48:30Z with timeout of 600
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:48:30Z with timeout of 600
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:48:30Z with timeout of 600
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:48:30Z with timeout of 600
>03:48:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:48:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:48:30Z with timeout of 600
>03:48:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 381 are : Creation time is 11.291316 , Creation speed 0.6376581790820486, Csi creation time is 11.248767
>03:48:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 382
>03:48:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:49:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 382
>03:49:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapi6q2s62t.yaml with timeout of 600
>03:49:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:49:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:49:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:49:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:49:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:49:06Z with timeout of 600
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:49:06Z with timeout of 600
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:49:06Z with timeout of 600
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:49:06Z with timeout of 600
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:49:06Z with timeout of 600
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:49:06Z with timeout of 600
>03:49:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 382 are : Creation time is 12.051485 , Creation speed 0.5974367474215834, Csi creation time is 12.013455
>03:49:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 383
>03:49:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:49:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 383
>03:49:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapop5hf8ar.yaml with timeout of 600
>03:49:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:49:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:49:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:49:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:49:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:50:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:49:41Z with timeout of 600
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:49:41Z with timeout of 600
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:49:41Z with timeout of 600
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:49:41Z with timeout of 600
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:49:41Z with timeout of 600
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:49:41Z with timeout of 600
>03:50:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 383 are : Creation time is 11.941438 , Creation speed 0.6029424596937153, Csi creation time is 11.902361
>03:50:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 384
>03:50:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:50:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 384
>03:50:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap890_pu1e.yaml with timeout of 600
>03:50:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:50:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:50:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:50:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:50:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:50:17Z with timeout of 600
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:50:17Z with timeout of 600
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:50:17Z with timeout of 600
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:50:17Z with timeout of 600
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:50:17Z with timeout of 600
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:50:17Z with timeout of 600
>03:50:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 384 are : Creation time is 12.204983 , Creation speed 0.5899229847350054, Csi creation time is 12.164821
>03:50:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 385
>03:50:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:50:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 385
>03:50:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwy5xub__.yaml with timeout of 600
>03:50:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:50:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:50:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:51:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:51:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:51:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:50:52Z with timeout of 600
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:50:52Z with timeout of 600
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:50:52Z with timeout of 600
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:50:52Z with timeout of 600
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:50:52Z with timeout of 600
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:50:52Z with timeout of 600
>03:51:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 385 are : Creation time is 12.773029 , Creation speed 0.5636877517462773, Csi creation time is 12.733366
>03:51:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 386
>03:51:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:51:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 386
>03:51:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8m_iltg4.yaml with timeout of 600
>03:51:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:51:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:51:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:51:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:51:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:51:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:51:27Z with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:51:27Z with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:51:27Z with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:51:27Z with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:51:27Z with timeout of 600
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:51:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:51:27Z with timeout of 600
>03:51:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 386 are : Creation time is 12.496232 , Creation speed 0.5761736817946402, Csi creation time is 12.455547
>03:51:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 387
>03:51:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:52:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 387
>03:52:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapr7efzu0u.yaml with timeout of 600
>03:52:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:52:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:52:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:52:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:52:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:52:02Z with timeout of 600
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:52:02Z with timeout of 600
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:52:02Z with timeout of 600
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:52:02Z with timeout of 600
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:52:02Z with timeout of 600
>03:52:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:52:02Z with timeout of 600
>03:52:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 387 are : Creation time is 12.31691 , Creation speed 0.5845621994477511, Csi creation time is 12.277907
>03:52:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 388
>03:52:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:52:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 388
>03:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapd4adb4lt.yaml with timeout of 600
>03:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:52:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:52:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:52:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:52:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:52:38Z with timeout of 600
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:52:38Z with timeout of 600
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:52:38Z with timeout of 600
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:52:38Z with timeout of 600
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:52:38Z with timeout of 600
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:52:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:52:38Z with timeout of 600
>03:52:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 388 are : Creation time is 12.069584 , Creation speed 0.5965408584090388, Csi creation time is 12.028651
>03:52:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 389
>03:52:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:53:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 389
>03:53:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapwfi68e4j.yaml with timeout of 600
>03:53:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:53:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:53:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:53:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:53:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:53:13Z with timeout of 600
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:53:13Z with timeout of 600
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:53:13Z with timeout of 600
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:53:13Z with timeout of 600
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:53:13Z with timeout of 600
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:53:13Z with timeout of 600
>03:53:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 389 are : Creation time is 11.991203 , Creation speed 0.6004401726832579, Csi creation time is 11.954548
>03:53:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 390
>03:53:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:53:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 390
>03:53:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap86zst9cu.yaml with timeout of 600
>03:53:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:53:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:53:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:53:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:53:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:54:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:54:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:54:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:54:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:53:48Z with timeout of 600
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:53:48Z with timeout of 600
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:53:48Z with timeout of 600
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:53:48Z with timeout of 600
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:53:48Z with timeout of 600
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:53:48Z with timeout of 600
>03:54:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 390 are : Creation time is 12.897206 , Creation speed 0.5582604480381255, Csi creation time is 12.857066
>03:54:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 391
>03:54:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:54:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 391
>03:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapcsxq2hag.yaml with timeout of 600
>03:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:54:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:54:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:54:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:54:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:54:24Z with timeout of 600
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:54:24Z with timeout of 600
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:54:24Z with timeout of 600
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:54:24Z with timeout of 600
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:54:24Z with timeout of 600
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:54:24Z with timeout of 600
>03:54:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 391 are : Creation time is 12.313014 , Creation speed 0.5847471626362156, Csi creation time is 12.267561
>03:54:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 392
>03:54:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:54:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 392
>03:54:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8h3xzlcp.yaml with timeout of 600
>03:54:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:54:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:54:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:55:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:55:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:55:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:54:59Z with timeout of 600
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:54:59Z with timeout of 600
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:54:59Z with timeout of 600
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:54:59Z with timeout of 600
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:54:59Z with timeout of 600
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:54:59Z with timeout of 600
>03:55:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 392 are : Creation time is 11.913339 , Creation speed 0.6043645698321856, Csi creation time is 11.873238
>03:55:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 393
>03:55:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:55:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 393
>03:55:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap5w73kc7k.yaml with timeout of 600
>03:55:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:55:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:55:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:55:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:55:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:55:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:55:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:55:34Z with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:55:34Z with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:55:34Z with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:55:34Z with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:55:34Z with timeout of 600
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:55:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:55:34Z with timeout of 600
>03:55:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 393 are : Creation time is 12.686757 , Creation speed 0.567520919648733, Csi creation time is 12.64398
>03:55:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 394
>03:55:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:56:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 394
>03:56:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapyaes_z1d.yaml with timeout of 600
>03:56:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:56:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:56:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:56:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:56:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:56:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:56:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:56:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:56:09Z with timeout of 600
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:56:09Z with timeout of 600
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:56:09Z with timeout of 600
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:56:09Z with timeout of 600
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:56:09Z with timeout of 600
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:56:09Z with timeout of 600
>03:56:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 394 are : Creation time is 11.867288 , Creation speed 0.6067098059809453, Csi creation time is 11.826429
>03:56:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 395
>03:56:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:56:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 395
>03:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3ax7gh8l.yaml with timeout of 600
>03:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:56:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:56:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:56:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:57:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:57:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:57:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:57:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:56:45Z with timeout of 600
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:56:45Z with timeout of 600
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:56:45Z with timeout of 600
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:56:45Z with timeout of 600
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:56:45Z with timeout of 600
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:56:45Z with timeout of 600
>03:57:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 395 are : Creation time is 11.938277 , Creation speed 0.6031021059404134, Csi creation time is 11.896339
>03:57:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 396
>03:57:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:57:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 396
>03:57:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapl4vrpi4z.yaml with timeout of 600
>03:57:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:57:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:57:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:57:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:57:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:57:22Z with timeout of 600
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:57:22Z with timeout of 600
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:57:22Z with timeout of 600
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:57:22Z with timeout of 600
>03:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:57:22Z with timeout of 600
>03:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:57:22Z with timeout of 600
>03:57:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 396 are : Creation time is 12.387334 , Creation speed 0.5812388686702079, Csi creation time is 12.344791
>03:57:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 397
>03:57:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:57:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 397
>03:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap36e443l8.yaml with timeout of 600
>03:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:57:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:57:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:58:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:58:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:57:56Z with timeout of 600
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:57:56Z with timeout of 600
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:57:56Z with timeout of 600
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:57:56Z with timeout of 600
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:57:56Z with timeout of 600
>03:58:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:57:56Z with timeout of 600
>03:58:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 397 are : Creation time is 12.331662 , Creation speed 0.5838629050974637, Csi creation time is 12.290859
>03:58:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 398
>03:58:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:58:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 398
>03:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapvc9b8irx.yaml with timeout of 600
>03:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:58:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:58:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:58:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:58:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:58:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:58:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:58:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:58:31Z with timeout of 600
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:58:31Z with timeout of 600
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:58:31Z with timeout of 600
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:58:31Z with timeout of 600
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:58:31Z with timeout of 600
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:58:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:58:31Z with timeout of 600
>03:58:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 398 are : Creation time is 12.716494 , Creation speed 0.566193795239474, Csi creation time is 12.674114
>03:58:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 399
>03:58:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:59:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 399
>03:59:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap1n490uju.yaml with timeout of 600
>03:59:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:59:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:59:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:59:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:59:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:59:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>03:59:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:59:06Z with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:59:06Z with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:59:06Z with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:59:06Z with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:59:06Z with timeout of 600
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:59:06Z with timeout of 600
>03:59:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 399 are : Creation time is 12.729895 , Creation speed 0.5655977523773762, Csi creation time is 12.686649
>03:59:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 400
>03:59:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>03:59:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 400
>03:59:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapu6t63inm.yaml with timeout of 600
>03:59:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:59:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>03:59:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>03:59:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>03:59:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:00:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:59:40Z with timeout of 600
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:59:40Z with timeout of 600
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T11:59:40Z with timeout of 600
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T11:59:40Z with timeout of 600
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T11:59:40Z with timeout of 600
>04:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T11:59:40Z with timeout of 600
>04:00:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 400 are : Creation time is 12.432211 , Creation speed 0.5791407497829629, Csi creation time is 12.394316
>04:00:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 401
>04:00:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:00:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 401
>04:00:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjrl0g1co.yaml with timeout of 600
>04:00:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:00:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:00:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:00:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:00:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:00:15Z with timeout of 600
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:00:15Z with timeout of 600
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:00:15Z with timeout of 600
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:00:15Z with timeout of 600
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:00:15Z with timeout of 600
>04:00:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:00:15Z with timeout of 600
>04:00:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 401 are : Creation time is 12.614618 , Creation speed 0.5707663918162247, Csi creation time is 12.573264
>04:00:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 402
>04:00:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:00:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 402
>04:00:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapl80djuk8.yaml with timeout of 600
>04:00:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:00:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:00:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:01:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:01:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:01:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:01:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:01:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:01:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:00:50Z with timeout of 600
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:00:50Z with timeout of 600
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:00:50Z with timeout of 600
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:00:50Z with timeout of 600
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:00:50Z with timeout of 600
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:00:50Z with timeout of 600
>04:01:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 402 are : Creation time is 13.055443 , Creation speed 0.5514941162854451, Csi creation time is 13.018761
>04:01:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 403
>04:01:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:01:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 403
>04:01:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjage_bjr.yaml with timeout of 600
>04:01:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:01:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:01:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:01:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:01:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:01:25Z with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:01:25Z with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:01:25Z with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:01:25Z with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:01:25Z with timeout of 600
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:01:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:01:25Z with timeout of 600
>04:01:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 403 are : Creation time is 13.459568 , Creation speed 0.5349354451792212, Csi creation time is 13.418223
>04:01:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 404
>04:01:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:02:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 404
>04:02:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapb1frckh2.yaml with timeout of 600
>04:02:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:02:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:02:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:02:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:02:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:02:00Z with timeout of 600
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:02:00Z with timeout of 600
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:02:00Z with timeout of 600
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:02:00Z with timeout of 600
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:02:00Z with timeout of 600
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:02:00Z with timeout of 600
>04:02:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 404 are : Creation time is 12.8404 , Creation speed 0.560730195321018, Csi creation time is 12.79895
>04:02:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 405
>04:02:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:02:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 405
>04:02:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapts64by8o.yaml with timeout of 600
>04:02:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:02:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:02:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:02:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:02:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:02:35Z with timeout of 600
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:02:35Z with timeout of 600
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:02:35Z with timeout of 600
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:02:35Z with timeout of 600
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:02:35Z with timeout of 600
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:02:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:02:35Z with timeout of 600
>04:02:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 405 are : Creation time is 12.847885 , Creation speed 0.5604035216691308, Csi creation time is 12.80556
>04:02:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 406
>04:02:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:03:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 406
>04:03:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapy34p6mqf.yaml with timeout of 600
>04:03:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:03:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:03:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:03:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:03:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:03:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:03:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:03:10Z with timeout of 600
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:03:10Z with timeout of 600
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:03:10Z with timeout of 600
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:03:10Z with timeout of 600
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:03:10Z with timeout of 600
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:03:10Z with timeout of 600
>04:03:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 406 are : Creation time is 12.940161 , Creation speed 0.5564072966325535, Csi creation time is 12.902176
>04:03:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 407
>04:03:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:03:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 407
>04:03:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0pqrb7k7.yaml with timeout of 600
>04:03:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:03:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:03:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:03:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:03:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:04:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:03:46Z with timeout of 600
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:03:46Z with timeout of 600
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:03:46Z with timeout of 600
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:03:46Z with timeout of 600
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:03:46Z with timeout of 600
>04:04:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:03:46Z with timeout of 600
>04:04:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 407 are : Creation time is 13.337374 , Creation speed 0.5398364025781986, Csi creation time is 13.295905
>04:04:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 408
>04:04:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:04:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 408
>04:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdb7zh_zz.yaml with timeout of 600
>04:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:04:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:04:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:04:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:04:23Z with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:04:23Z with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:04:23Z with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:04:23Z with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:04:23Z with timeout of 600
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:04:23Z with timeout of 600
>04:04:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 408 are : Creation time is 13.537171 , Creation speed 0.5318688816149253, Csi creation time is 13.496162
>04:04:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 409
>04:04:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:04:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 409
>04:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaprj6rhezn.yaml with timeout of 600
>04:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:04:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:04:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:05:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:05:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:05:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:04:58Z with timeout of 600
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:04:58Z with timeout of 600
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:04:58Z with timeout of 600
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:04:58Z with timeout of 600
>04:05:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:05:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:05:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:04:58Z with timeout of 600
>04:05:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:04:58Z with timeout of 600
>04:05:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 409 are : Creation time is 14.237208 , Creation speed 0.5057171321792868, Csi creation time is 14.195672
>04:05:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 410
>04:05:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:05:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 410
>04:05:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapi6id8tiw.yaml with timeout of 600
>04:05:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:05:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:05:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:05:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:05:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:05:33Z with timeout of 600
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:05:33Z with timeout of 600
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:05:33Z with timeout of 600
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:05:33Z with timeout of 600
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:05:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:05:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:05:33Z with timeout of 600
>04:05:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:05:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:05:33Z with timeout of 600
>04:05:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 410 are : Creation time is 14.37432 , Creation speed 0.5008932596463693, Csi creation time is 14.329078
>04:05:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 411
>04:05:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:06:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 411
>04:06:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapi2svpmwr.yaml with timeout of 600
>04:06:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:06:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:06:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:06:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:06:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:06:07Z with timeout of 600
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:06:07Z with timeout of 600
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:06:07Z with timeout of 600
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:06:07Z with timeout of 600
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:06:07Z with timeout of 600
>04:06:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:06:07Z with timeout of 600
>04:06:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 411 are : Creation time is 13.721419 , Creation speed 0.5247270708663587, Csi creation time is 13.680706
>04:06:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 412
>04:06:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:06:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 412
>04:06:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapv6_5b4wx.yaml with timeout of 600
>04:06:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:06:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:06:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:06:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:06:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:07:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:06:42Z with timeout of 600
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:06:42Z with timeout of 600
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:06:42Z with timeout of 600
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:06:42Z with timeout of 600
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:06:42Z with timeout of 600
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:06:42Z with timeout of 600
>04:07:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 412 are : Creation time is 14.074173 , Creation speed 0.5115753515322001, Csi creation time is 14.033915
>04:07:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 413
>04:07:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:07:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 413
>04:07:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0u8hrrhh.yaml with timeout of 600
>04:07:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:07:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:07:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:07:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:07:17Z with timeout of 600
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:07:17Z with timeout of 600
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:07:17Z with timeout of 600
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:07:17Z with timeout of 600
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:07:17Z with timeout of 600
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:07:17Z with timeout of 600
>04:07:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 413 are : Creation time is 14.130998 , Creation speed 0.5095181529287599, Csi creation time is 14.089183
>04:07:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 414
>04:07:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:07:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 414
>04:07:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjcb4gda4.yaml with timeout of 600
>04:07:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:07:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:07:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:08:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:08:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:08:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:07:52Z with timeout of 600
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:07:52Z with timeout of 600
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:07:52Z with timeout of 600
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:07:52Z with timeout of 600
>04:08:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:08:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:08:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:07:52Z with timeout of 600
>04:08:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:07:52Z with timeout of 600
>04:08:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 414 are : Creation time is 15.202575 , Creation speed 0.47360397827341755, Csi creation time is 15.162668
>04:08:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 415
>04:08:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:08:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 415
>04:08:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap4je1qd1g.yaml with timeout of 600
>04:08:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:08:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:08:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:08:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:08:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:08:27Z with timeout of 600
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:08:27Z with timeout of 600
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:08:27Z with timeout of 600
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:08:27Z with timeout of 600
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:08:27Z with timeout of 600
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:08:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:08:27Z with timeout of 600
>04:08:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 415 are : Creation time is 15.190323 , Creation speed 0.47398597120021746, Csi creation time is 15.152387
>04:08:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 416
>04:08:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:09:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 416
>04:09:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap_rkt6jw5.yaml with timeout of 600
>04:09:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:09:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:09:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:09:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:09:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:09:02Z with timeout of 600
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:09:02Z with timeout of 600
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:09:02Z with timeout of 600
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:09:02Z with timeout of 600
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:09:02Z with timeout of 600
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:09:02Z with timeout of 600
>04:09:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 416 are : Creation time is 15.095445 , Creation speed 0.4769650712516259, Csi creation time is 15.055546
>04:09:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 417
>04:09:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:09:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 417
>04:09:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snaptnis529s.yaml with timeout of 600
>04:09:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:09:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:09:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:09:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:09:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:09:37Z with timeout of 600
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:09:37Z with timeout of 600
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:09:37Z with timeout of 600
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:09:37Z with timeout of 600
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:09:37Z with timeout of 600
>04:09:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:09:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:09:37Z with timeout of 600
>04:09:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 417 are : Creation time is 14.756488 , Creation speed 0.48792097415048896, Csi creation time is 14.710871
>04:09:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 418
>04:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:10:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 418
>04:10:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapokg18ycf.yaml with timeout of 600
>04:10:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:10:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:10:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:10:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:10:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:10:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:10:15Z with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:10:15Z with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:10:15Z with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:10:15Z with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:10:15Z with timeout of 600
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:10:15Z with timeout of 600
>04:10:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 418 are : Creation time is 15.590141 , Creation speed 0.4618303323876289, Csi creation time is 15.551665
>04:10:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 419
>04:10:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:10:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 419
>04:10:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmu09fmsc.yaml with timeout of 600
>04:10:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:10:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:10:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:11:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:11:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:11:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:10:51Z with timeout of 600
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:10:51Z with timeout of 600
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:10:51Z with timeout of 600
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:10:51Z with timeout of 600
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:10:51Z with timeout of 600
>04:11:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:10:51Z with timeout of 600
>04:11:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 419 are : Creation time is 15.434293 , Creation speed 0.46649367094430566, Csi creation time is 15.394759
>04:11:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 420
>04:11:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:11:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 420
>04:11:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3vfjw38i.yaml with timeout of 600
>04:11:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:11:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:11:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:11:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:11:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:11:27Z with timeout of 600
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:11:27Z with timeout of 600
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:11:27Z with timeout of 600
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:11:27Z with timeout of 600
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:11:27Z with timeout of 600
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:11:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:11:27Z with timeout of 600
>04:11:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 420 are : Creation time is 15.120558 , Creation speed 0.4761729031428602, Csi creation time is 15.074994
>04:11:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 421
>04:11:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:12:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 421
>04:12:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapxa0xlvmi.yaml with timeout of 600
>04:12:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:12:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:12:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:12:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:12:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:12:02Z with timeout of 600
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:12:02Z with timeout of 600
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:12:02Z with timeout of 600
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:12:02Z with timeout of 600
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:12:02Z with timeout of 600
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:12:02Z with timeout of 600
>04:12:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 421 are : Creation time is 15.263159 , Creation speed 0.4717241037717028, Csi creation time is 15.221467
>04:12:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 422
>04:12:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:12:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 422
>04:12:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap0_6orzlq.yaml with timeout of 600
>04:12:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:12:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:12:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:12:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:12:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:12:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:12:36Z with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:12:36Z with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:12:36Z with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:12:36Z with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:12:36Z with timeout of 600
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:12:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:12:36Z with timeout of 600
>04:12:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 422 are : Creation time is 15.882825 , Creation speed 0.45331985965972676, Csi creation time is 15.83912
>04:12:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 423
>04:12:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:13:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 423
>04:13:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapilqf64eu.yaml with timeout of 600
>04:13:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:13:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:13:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:13:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:13:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:13:10Z with timeout of 600
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:13:10Z with timeout of 600
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:13:10Z with timeout of 600
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:13:10Z with timeout of 600
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:13:10Z with timeout of 600
>04:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:13:10Z with timeout of 600
>04:13:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 423 are : Creation time is 15.675803 , Creation speed 0.4593066141492082, Csi creation time is 15.63641
>04:13:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 424
>04:13:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:13:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 424
>04:13:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapyrh3yj1z.yaml with timeout of 600
>04:13:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:13:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:13:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:13:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:13:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:14:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:13:45Z with timeout of 600
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:13:45Z with timeout of 600
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:13:45Z with timeout of 600
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:13:45Z with timeout of 600
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:13:45Z with timeout of 600
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:13:45Z with timeout of 600
>04:14:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 424 are : Creation time is 15.974903 , Creation speed 0.45070696203914357, Csi creation time is 15.914744
>04:14:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 425
>04:14:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:14:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 425
>04:14:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3oes9yz5.yaml with timeout of 600
>04:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:14:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:14:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:14:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:14:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:14:19Z with timeout of 600
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:14:19Z with timeout of 600
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:14:19Z with timeout of 600
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:14:19Z with timeout of 600
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:14:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:14:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:14:19Z with timeout of 600
>04:14:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:14:19Z with timeout of 600
>04:14:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 425 are : Creation time is 16.47349 , Creation speed 0.4370658555048141, Csi creation time is 16.432017
>04:14:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 426
>04:14:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:14:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 426
>04:14:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapc8656sl2.yaml with timeout of 600
>04:14:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:14:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:14:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:15:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:15:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:15:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:14:55Z with timeout of 600
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:14:55Z with timeout of 600
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:14:55Z with timeout of 600
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:14:55Z with timeout of 600
>04:15:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:15:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:15:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:14:55Z with timeout of 600
>04:15:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:14:55Z with timeout of 600
>04:15:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 426 are : Creation time is 16.337731 , Creation speed 0.4406976709311715, Csi creation time is 16.296615
>04:15:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 427
>04:15:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:15:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 427
>04:15:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7m8ngkwz.yaml with timeout of 600
>04:15:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:31 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:15:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:15:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:15:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:15:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:15:31Z with timeout of 600
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:15:31Z with timeout of 600
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:15:31Z with timeout of 600
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:15:31Z with timeout of 600
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:15:31Z with timeout of 600
>04:15:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:15:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:15:31Z with timeout of 600
>04:15:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 427 are : Creation time is 16.542564 , Creation speed 0.4352408731802398, Csi creation time is 16.505005
>04:15:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 428
>04:15:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:16:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 428
>04:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap10lokynv.yaml with timeout of 600
>04:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:16:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:16:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:16:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:16:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:16:07Z with timeout of 600
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:16:07Z with timeout of 600
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:16:07Z with timeout of 600
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:16:07Z with timeout of 600
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:16:07Z with timeout of 600
>04:16:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:16:07Z with timeout of 600
>04:16:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 428 are : Creation time is 17.458805 , Creation speed 0.4123993595208836, Csi creation time is 17.417252
>04:16:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 429
>04:16:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:16:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 429
>04:16:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7yz6h13y.yaml with timeout of 600
>04:16:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:16:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:16:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:16:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:16:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:17:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:17:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:16:43Z with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:16:43Z with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:16:43Z with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:16:43Z with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:16:43Z with timeout of 600
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:16:43Z with timeout of 600
>04:17:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 429 are : Creation time is 16.657159 , Creation speed 0.4322465793836752, Csi creation time is 16.614462
>04:17:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 430
>04:17:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:17:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 430
>04:17:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapg0q221k3.yaml with timeout of 600
>04:17:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:17:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:17:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:17:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:17:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:17:20Z with timeout of 600
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:17:20Z with timeout of 600
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:17:20Z with timeout of 600
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:17:20Z with timeout of 600
>04:17:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:17:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:17:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:17:20Z with timeout of 600
>04:17:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:17:20Z with timeout of 600
>04:17:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 430 are : Creation time is 16.315766 , Creation speed 0.44129095747021624, Csi creation time is 16.275561
>04:17:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 431
>04:17:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:17:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 431
>04:17:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapi84__ns0.yaml with timeout of 600
>04:17:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:17:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:17:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:18:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:18:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:17:54Z with timeout of 600
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:17:54Z with timeout of 600
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:17:54Z with timeout of 600
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:17:54Z with timeout of 600
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:17:54Z with timeout of 600
>04:18:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:17:54Z with timeout of 600
>04:18:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 431 are : Creation time is 16.613451 , Creation speed 0.4333837683693773, Csi creation time is 16.575934
>04:18:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 432
>04:18:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:18:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 432
>04:18:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap7amzwf1o.yaml with timeout of 600
>04:18:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:30 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:18:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:18:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:18:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:18:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:18:30Z with timeout of 600
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:18:30Z with timeout of 600
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:18:30Z with timeout of 600
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:18:30Z with timeout of 600
>04:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:18:30Z with timeout of 600
>04:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:18:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:18:30Z with timeout of 600
>04:18:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 432 are : Creation time is 16.326967 , Creation speed 0.44098821293630347, Csi creation time is 16.284252
>04:18:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 433
>04:18:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:19:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 433
>04:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap3cv6qujs.yaml with timeout of 600
>04:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:19:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:19:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:19:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:19:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:19:05Z with timeout of 600
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:19:05Z with timeout of 600
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:19:05Z with timeout of 600
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:19:05Z with timeout of 600
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:19:05Z with timeout of 600
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:19:05Z with timeout of 600
>04:19:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 433 are : Creation time is 16.080917 , Creation speed 0.44773566084570926, Csi creation time is 16.038574
>04:19:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 434
>04:19:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:19:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 434
>04:19:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapw4yswohe.yaml with timeout of 600
>04:19:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:19:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:19:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:19:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:19:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:20:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:19:40Z with timeout of 600
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:19:40Z with timeout of 600
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:19:40Z with timeout of 600
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:19:40Z with timeout of 600
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:19:40Z with timeout of 600
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:19:40Z with timeout of 600
>04:20:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 434 are : Creation time is 16.311014 , Creation speed 0.441419521802875, Csi creation time is 16.273019
>04:20:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 435
>04:20:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:20:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 435
>04:20:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapo7i8ar0_.yaml with timeout of 600
>04:20:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:20:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:20:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:20:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:20:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:20:14Z with timeout of 600
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:20:14Z with timeout of 600
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:20:14Z with timeout of 600
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:20:14Z with timeout of 600
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:20:14Z with timeout of 600
>04:20:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:20:14Z with timeout of 600
>04:20:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 435 are : Creation time is 16.60303 , Creation speed 0.43365578451644066, Csi creation time is 16.557691
>04:20:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 436
>04:20:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:20:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 436
>04:20:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbq5vs9yc.yaml with timeout of 600
>04:20:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:20:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:20:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:20:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:21:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:21:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:21:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:21:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:20:49Z with timeout of 600
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:20:49Z with timeout of 600
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:20:49Z with timeout of 600
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:20:49Z with timeout of 600
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:20:49Z with timeout of 600
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:20:49Z with timeout of 600
>04:21:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 436 are : Creation time is 16.968654 , Creation speed 0.4243117927915791, Csi creation time is 16.925679
>04:21:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 437
>04:21:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:21:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 437
>04:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapmo7sdxsr.yaml with timeout of 600
>04:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:21:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:21:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:21:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:21:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:21:24Z with timeout of 600
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:21:24Z with timeout of 600
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:21:24Z with timeout of 600
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:21:24Z with timeout of 600
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:21:24Z with timeout of 600
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:21:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:21:24Z with timeout of 600
>04:21:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 437 are : Creation time is 16.954014 , Creation speed 0.4246781912531156, Csi creation time is 16.915264
>04:21:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 438
>04:21:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:22:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 438
>04:22:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapjyruzc7k.yaml with timeout of 600
>04:22:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:22:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:22:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:22:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:22:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:22:00Z with timeout of 600
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:22:00Z with timeout of 600
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:22:00Z with timeout of 600
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:22:00Z with timeout of 600
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:22:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:22:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:22:00Z with timeout of 600
>04:22:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:22:00Z with timeout of 600
>04:22:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 438 are : Creation time is 17.491733 , Creation speed 0.41162302214423235, Csi creation time is 17.451783
>04:22:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 439
>04:22:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:22:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 439
>04:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapbebcgs2n.yaml with timeout of 600
>04:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:22:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:22:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:22:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:22:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:22:36Z with timeout of 600
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:22:36Z with timeout of 600
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:22:36Z with timeout of 600
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:22:36Z with timeout of 600
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:22:36Z with timeout of 600
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:22:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:22:36Z with timeout of 600
>04:22:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 439 are : Creation time is 18.391693 , Creation speed 0.39148108877197985, Csi creation time is 18.349302
>04:22:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 440
>04:22:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:23:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 440
>04:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapd4ff5hty.yaml with timeout of 600
>04:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:23:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:23:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:23:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:23:12Z with timeout of 600
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:23:12Z with timeout of 600
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:23:12Z with timeout of 600
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:23:12Z with timeout of 600
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:23:12Z with timeout of 600
>04:23:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:23:12Z with timeout of 600
>04:23:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 440 are : Creation time is 17.51784 , Creation speed 0.411009576523133, Csi creation time is 17.477558
>04:23:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 441
>04:23:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:23:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 441
>04:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapy0dzhf6t.yaml with timeout of 600
>04:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:23:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:23:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:23:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:23:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:24:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:23:47Z with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:23:47Z with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:23:47Z with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:23:47Z with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:23:47Z with timeout of 600
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:23:47Z with timeout of 600
>04:24:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 441 are : Creation time is 17.799482 , Creation speed 0.40450615360604314, Csi creation time is 17.755006
>04:24:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 442
>04:24:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:24:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 442
>04:24:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapch34ytzh.yaml with timeout of 600
>04:24:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:24:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:24:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:24:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:24:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:24:22Z with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:24:22Z with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:24:22Z with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:24:22Z with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:24:22Z with timeout of 600
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:24:22Z with timeout of 600
>04:24:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 442 are : Creation time is 17.662217 , Creation speed 0.40764984373139573, Csi creation time is 17.621388
>04:24:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 443
>04:24:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:24:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 443
>04:24:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap2yyrfi13.yaml with timeout of 600
>04:24:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:24:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:24:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:25:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:25:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:25:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:24:57Z with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:24:57Z with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:24:57Z with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:24:57Z with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:24:57Z with timeout of 600
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:24:57Z with timeout of 600
>04:25:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 443 are : Creation time is 17.827299 , Creation speed 0.4038749784810363, Csi creation time is 17.788136
>04:25:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 444
>04:25:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:25:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 444
>04:25:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapsoduwtwz.yaml with timeout of 600
>04:25:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:32 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:25:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:25:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:42 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:25:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:25:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:52 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:25:32Z with timeout of 600
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:25:32Z with timeout of 600
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:25:32Z with timeout of 600
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:25:32Z with timeout of 600
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:25:32Z with timeout of 600
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:25:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:25:32Z with timeout of 600
>04:25:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 444 are : Creation time is 17.917864 , Creation speed 0.40183361141707513, Csi creation time is 17.876998
>04:25:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 445
>04:25:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:26:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 445
>04:26:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap8trlq9rs.yaml with timeout of 600
>04:26:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:26:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:26:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:26:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:26:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:26:07Z with timeout of 600
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:26:07Z with timeout of 600
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:26:07Z with timeout of 600
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:26:07Z with timeout of 600
>04:26:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:26:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:26:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:26:07Z with timeout of 600
>04:26:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:26:07Z with timeout of 600
>04:26:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 445 are : Creation time is 18.579053 , Creation speed 0.3875332074245119, Csi creation time is 18.536096
>04:26:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 446
>04:26:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:26:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 446
>04:26:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapeut7ykk_.yaml with timeout of 600
>04:26:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:41 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:26:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:26:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:26:51 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:26:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:27:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:01 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:27:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:26:41Z with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:26:41Z with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:26:41Z with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:26:41Z with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:26:41Z with timeout of 600
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:02 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:26:41Z with timeout of 600
>04:27:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 446 are : Creation time is 19.722714 , Creation speed 0.36506131965408006, Csi creation time is 19.68671
>04:27:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 447
>04:27:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:27:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 447
>04:27:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapefe2ie18.yaml with timeout of 600
>04:27:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:27:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:27:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:27:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:27:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:27:16Z with timeout of 600
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:27:16Z with timeout of 600
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:27:16Z with timeout of 600
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:27:16Z with timeout of 600
>04:27:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:27:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:27:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:27:16Z with timeout of 600
>04:27:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:27:16Z with timeout of 600
>04:27:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 447 are : Creation time is 19.532372 , Creation speed 0.3686188241755789, Csi creation time is 19.491452
>04:27:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 448
>04:27:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:27:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 448
>04:27:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapczx5_czd.yaml with timeout of 600
>04:27:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:27:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:27:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:28:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:28:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:28:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:27:50Z with timeout of 600
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:27:50Z with timeout of 600
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:27:50Z with timeout of 600
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:27:50Z with timeout of 600
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:11 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:27:50Z with timeout of 600
>04:28:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:27:50Z with timeout of 600
>04:28:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 448 are : Creation time is 17.847268 , Creation speed 0.4034230897412422, Csi creation time is 17.807277
>04:28:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 449
>04:28:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:28:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 449
>04:28:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapdiu5a31o.yaml with timeout of 600
>04:28:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:28:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:28:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:28:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:28:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:28:25Z with timeout of 600
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:28:25Z with timeout of 600
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:28:25Z with timeout of 600
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:28:25Z with timeout of 600
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:28:25Z with timeout of 600
>04:28:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:28:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:28:25Z with timeout of 600
>04:28:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 449 are : Creation time is 18.865879 , Creation speed 0.38164137488637556, Csi creation time is 18.823125
>04:28:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 450
>04:28:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:29:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 450
>04:29:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snap98en15sk.yaml with timeout of 600
>04:29:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:29:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:29:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:29:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:29:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:20 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:29:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:29:00Z with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:29:00Z with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:29:00Z with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:29:00Z with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:29:00Z with timeout of 600
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:21 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:29:00Z with timeout of 600
>04:29:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 450 are : Creation time is 18.968184 , Creation speed 0.37958299012704644, Csi creation time is 18.930099
>04:29:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 451
>04:29:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:29:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 451
>04:29:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapw4uftwu1.yaml with timeout of 600
>04:29:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:29:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:29:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:29:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:29:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3 Created and ready to use
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:29:38Z with timeout of 600
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:29:38Z with timeout of 600
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator get pod with timeout of 600
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-4qdlq --since-time=2022-11-29T12:29:38Z with timeout of 600
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-cluster-storage-operator logs csi-snapshot-controller-7c4f69d8cd-bhtcb --since-time=2022-11-29T12:29:38Z with timeout of 600
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage get pod with timeout of 600
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - The logs pods are : ['csi-rbdplugin-provisioner-67c84f6f8c-m4lg6', 'csi-rbdplugin-provisioner-67c84f6f8c-mdvmg']
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-m4lg6 -c csi-rbdplugin --since-time=2022-11-29T12:29:38Z with timeout of 600
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:29:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage logs csi-rbdplugin-provisioner-67c84f6f8c-mdvmg -c csi-rbdplugin --since-time=2022-11-29T12:29:38Z with timeout of 600
>04:29:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Results for snapshot number 451 are : Creation time is 18.359025 , Creation speed 0.3921776891746703, Csi creation time is 18.318252
>04:29:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Starting test number 452
>04:29:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 rsh pod-test-rbd-2f1314eacb67426f8ceded15548 fio --name=fio-fillup --rw=write --bs=4m --direct=1 --numjobs=1 --time_based=0 --runtime=36000 --ioengine=libaio --end_fsync=1 --filename=/mnt/pod-test-rbd-2f1314eacb67426f8ceded15548 --size=7372M --output-format=json
>04:30:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Taking snapshot number 452
>04:30:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:30:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 create -f /tmp/Snapqimqzfiz.yaml with timeout of 600
>04:30:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:30:12 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:30:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:30:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:30:22 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:30:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:30:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:30:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:30:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:30:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:30:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:30:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:30:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:30:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:31:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:31:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:31:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:31:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:31:13 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:31:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:31:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:31:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:31:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:31:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:31:33 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:31:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:31:43 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:31:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:31:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:31:53 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:31:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:32:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:32:03 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:32:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:32:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:32:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:32:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:32:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:32:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:32:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:32:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:32:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:32:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:32:44 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:32:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:32:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:32:54 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:32:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:33:04 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:33:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:33:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:33:14 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:33:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:33:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:33:24 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:33:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:33:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:33:34 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:33:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:33:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:33:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:33:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:33:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:34:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:34:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:34:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:34:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:34:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:34:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:34:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:34:25 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:34:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:34:35 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:34:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:34:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:34:45 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:34:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:34:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:34:55 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:34:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:35:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:35:05 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:35:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:35:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:35:15 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:35:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:35:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:35:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:35:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:35:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:35:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:35:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:35:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:35:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:35:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:35:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:35:56 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:35:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:36:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:36:06 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:36:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:36:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:36:16 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:36:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:36:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:36:26 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:36:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:36:36 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:36:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:36:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:36:46 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:36:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:36:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:36:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:36:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:37:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:37:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:37:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:37:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:37:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:37:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:37:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:37:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:37:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:37:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:37:37 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:37:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:37:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:37:47 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:37:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:37:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:37:57 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:37:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:38:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:38:07 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:38:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:38:17 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:38:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:38:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:38:27 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:38:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:38:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:38:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:38:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:38:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:38:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:38:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:38:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:38:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:38:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:39:08 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:39:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:39:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:39:18 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:39:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:39:28 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:39:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:39:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:39:38 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:39:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:39:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:39:48 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:39:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:39:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:39:58 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:39:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:40:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:40:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:40:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:40:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:40:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:40:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:40:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:40:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:40:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:40:39 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:40:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:40:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:40:49 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:40:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:40:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:40:59 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:40:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:41:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:41:09 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:41:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:41:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:41:19 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:41:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:41:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:41:29 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:41:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:41:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:41:40 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:41:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:41:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:41:50 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:41:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:42:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:42:00 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:42:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:42:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>04:42:10 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 -o yaml with timeout of 600
>04:42:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 is not ready yet, sleep 10 sec before re-check
>04:42:20 - MainThread - test_pvc_multi_snapshot_performance - ERROR  - Snapshot was not created on time
>04:42:20 - MainThread - ocs_ci.helpers.helpers - INFO  - Creating directory /tmp/failed_testcase_ocs_logs_1669712909/test_pvc_multiple_snapshot_performance[CephBlockPool-512]_performance_stats
>04:42:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephCluster  -n openshift-storage -o yaml
>04:42:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephFilesystem  -n openshift-storage -o yaml
>04:42:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephBlockPool  -n openshift-storage -o yaml
>04:42:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>04:42:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-mon -o yaml
>04:42:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-mon-a-7885556b8d-nfw4h -n openshift-storage
>04:42:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>04:42:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-mon-b-56696c6688-jk7m8 -n openshift-storage
>04:42:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>04:42:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-mon-c-655dfcd8b8-8b6wg -n openshift-storage
>04:42:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>04:42:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-mds -o yaml
>04:42:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-mgr -o yaml
>04:42:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-osd -o yaml
>04:42:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=noobaa -o yaml
>04:42:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-rgw -o yaml
>04:42:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>04:42:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>04:42:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>04:42:32 - MainThread - ocs_ci.ocs.cluster - INFO  - port=3300
>04:42:32 - MainThread - ocs_ci.ocs.cluster - INFO  - port=3300
>04:42:32 - MainThread - ocs_ci.ocs.cluster - INFO  - port=3300
>04:42:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephCluster ocs-storagecluster-cephcluster -n openshift-storage -o yaml
>04:42:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get CephFilesystem ocs-storagecluster-cephfilesystem -n openshift-storage -o yaml
>04:42:33 - MainThread - ocs_ci.ocs.cluster - INFO  - Number of mons = 3
>04:42:33 - MainThread - ocs_ci.ocs.cluster - INFO  - Number of mds = 2
>04:42:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get StorageCluster ocs-storagecluster -n openshift-storage -o yaml
>04:42:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>04:42:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod rook-ceph-tools-749dfcffb9-jr87p -n openshift-storage
>04:42:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage -o yaml
>04:42:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph status --format json-pretty
>04:42:35 - MainThread - ocs_ci.ocs.cluster - INFO  - The IOPS percentage of the cluster is 4.55%
>04:42:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph status
>04:42:36 - MainThread - ocs_ci.ocs.cluster - INFO  - The RD throughput is 0.594726258 MiB/s
>04:42:36 - MainThread - ocs_ci.ocs.cluster - INFO  - The WR throughput is 0.603515316 MiB/s
>04:42:36 - MainThread - ocs_ci.ocs.cluster - INFO  - The throughput percentage of the cluster is 0.24140612639999998%
>04:42:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>04:42:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master0.ocp.openshift.lan
>04:42:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master1.ocp.openshift.lan
>04:42:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master2.ocp.openshift.lan
>04:42:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker1.ocp.openshift.lan
>04:42:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker2.ocp.openshift.lan
>04:42:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker3.ocp.openshift.lan
>04:42:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:38 - MainThread - ocs_ci.ocs.node - INFO  - Waiting for nodes ['smc-conv-master0.ocp.openshift.lan', 'smc-conv-master1.ocp.openshift.lan', 'smc-conv-master2.ocp.openshift.lan'] to reach status Ready
>04:42:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>04:42:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master0.ocp.openshift.lan
>04:42:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:39 - MainThread - ocs_ci.ocs.node - INFO  - Node smc-conv-master0.ocp.openshift.lan reached status Ready
>04:42:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master1.ocp.openshift.lan
>04:42:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:39 - MainThread - ocs_ci.ocs.node - INFO  - Node smc-conv-master1.ocp.openshift.lan reached status Ready
>04:42:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master2.ocp.openshift.lan
>04:42:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:39 - MainThread - ocs_ci.ocs.node - INFO  - Node smc-conv-master2.ocp.openshift.lan reached status Ready
>04:42:39 - MainThread - ocs_ci.ocs.node - INFO  - The following nodes reached status Ready: ['smc-conv-master0.ocp.openshift.lan', 'smc-conv-master1.ocp.openshift.lan', 'smc-conv-master2.ocp.openshift.lan']
>04:42:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc adm top nodes
>04:42:40 - MainThread - ocs_ci.ocs.node - INFO  - The CPU utilized by the node smc-conv-master0.ocp.openshift.lan is 8%
>04:42:40 - MainThread - ocs_ci.ocs.node - INFO  - The memory utilized of the node smc-conv-master0.ocp.openshift.lan is 10%
>04:42:40 - MainThread - ocs_ci.ocs.node - INFO  - The CPU utilized by the node smc-conv-master1.ocp.openshift.lan is 2%
>04:42:40 - MainThread - ocs_ci.ocs.node - INFO  - The memory utilized of the node smc-conv-master1.ocp.openshift.lan is 9%
>04:42:40 - MainThread - ocs_ci.ocs.node - INFO  - The CPU utilized by the node smc-conv-master2.ocp.openshift.lan is 1%
>04:42:40 - MainThread - ocs_ci.ocs.node - INFO  - The memory utilized of the node smc-conv-master2.ocp.openshift.lan is 10%
>04:42:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>04:42:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master0.ocp.openshift.lan
>04:42:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master1.ocp.openshift.lan
>04:42:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master2.ocp.openshift.lan
>04:42:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker1.ocp.openshift.lan
>04:42:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker2.ocp.openshift.lan
>04:42:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker3.ocp.openshift.lan
>04:42:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:42 - MainThread - ocs_ci.ocs.node - INFO  - Waiting for nodes ['smc-conv-worker1.ocp.openshift.lan', 'smc-conv-worker2.ocp.openshift.lan', 'smc-conv-worker3.ocp.openshift.lan'] to reach status Ready
>04:42:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>04:42:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker1.ocp.openshift.lan
>04:42:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - Node smc-conv-worker1.ocp.openshift.lan reached status Ready
>04:42:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker2.ocp.openshift.lan
>04:42:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - Node smc-conv-worker2.ocp.openshift.lan reached status Ready
>04:42:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker3.ocp.openshift.lan
>04:42:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - Node smc-conv-worker3.ocp.openshift.lan reached status Ready
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - The following nodes reached status Ready: ['smc-conv-worker1.ocp.openshift.lan', 'smc-conv-worker2.ocp.openshift.lan', 'smc-conv-worker3.ocp.openshift.lan']
>04:42:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc adm top nodes
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - The CPU utilized by the node smc-conv-worker1.ocp.openshift.lan is 7%
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - The memory utilized of the node smc-conv-worker1.ocp.openshift.lan is 4%
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - The CPU utilized by the node smc-conv-worker2.ocp.openshift.lan is 0%
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - The memory utilized of the node smc-conv-worker2.ocp.openshift.lan is 3%
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - The CPU utilized by the node smc-conv-worker3.ocp.openshift.lan is 0%
>04:42:43 - MainThread - ocs_ci.ocs.node - INFO  - The memory utilized of the node smc-conv-worker3.ocp.openshift.lan is 5%
>04:42:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>04:42:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master0.ocp.openshift.lan
>04:42:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master1.ocp.openshift.lan
>04:42:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master2.ocp.openshift.lan
>04:42:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker1.ocp.openshift.lan
>04:42:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker2.ocp.openshift.lan
>04:42:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker3.ocp.openshift.lan
>04:42:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc describe node smc-conv-master0.ocp.openshift.lan
>04:42:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc describe node smc-conv-master1.ocp.openshift.lan
>04:42:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc describe node smc-conv-master2.ocp.openshift.lan
>04:42:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get node  -o yaml
>04:42:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master0.ocp.openshift.lan
>04:42:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master1.ocp.openshift.lan
>04:42:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-master2.ocp.openshift.lan
>04:42:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker1.ocp.openshift.lan
>04:42:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker2.ocp.openshift.lan
>04:42:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node smc-conv-worker3.ocp.openshift.lan
>04:42:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>04:42:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc describe node smc-conv-worker1.ocp.openshift.lan
>04:42:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc describe node smc-conv-worker2.ocp.openshift.lan
>04:42:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc describe node smc-conv-worker3.ocp.openshift.lan
>FAILED
>------------------------------ live log teardown -------------------------------
>04:42:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Cleanup the test environment
>04:42:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>04:42:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get volumesnapshot  -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>04:42:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Trying to delete all (452) Snapshots
>04:42:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-1-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-1-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-10-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-10-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-100-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-100-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-101-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-101-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-102-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-102-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-103-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-103-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-104-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-104-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-105-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-105-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-106-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:42:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-106-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-107-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-107-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-108-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-108-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-109-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-109-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-11-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-11-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-110-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-110-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-111-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-111-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-112-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-112-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-113-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-113-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-114-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-114-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-115-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-115-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-116-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-116-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-117-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-117-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-118-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-118-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-119-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-119-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-12-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-12-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-120-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-120-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-121-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-121-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-122-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-122-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-123-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-123-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-124-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-124-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-125-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-125-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-126-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-126-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-127-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-127-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-128-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-128-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-129-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-129-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-13-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-13-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-130-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-130-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-131-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-131-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-132-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-132-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-133-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-133-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-134-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-134-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-135-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-135-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-136-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-136-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-137-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-137-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-138-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-138-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-139-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-139-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-14-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-14-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-140-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-140-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-141-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-141-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-142-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-142-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-143-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-143-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-144-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-144-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-145-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-145-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-146-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-146-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-147-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-147-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-148-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-148-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-149-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-149-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-15-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-15-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-150-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-150-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-151-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-151-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-152-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-152-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-153-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-153-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-154-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-154-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-155-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-155-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-156-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-156-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-157-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-157-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-158-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-158-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-159-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-159-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-16-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:43:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-16-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-160-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-160-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-161-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-161-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-162-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-162-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-163-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-163-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-164-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-164-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-165-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-165-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-166-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-166-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-167-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-167-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-168-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-168-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-169-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-169-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-17-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-17-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-170-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-170-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-171-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-171-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-172-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-172-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-173-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-173-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-174-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-174-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-175-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-175-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-176-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-176-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-177-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-177-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-178-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-178-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-179-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-179-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-18-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-18-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-180-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-180-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-181-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-181-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-182-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-182-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-183-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-183-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-184-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-184-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-185-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-185-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-186-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-186-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-187-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-187-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-188-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-188-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-189-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-189-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-19-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-19-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-190-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-190-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-191-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-191-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-192-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-192-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-193-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-193-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-194-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-194-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-195-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-195-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-196-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-196-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-197-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-197-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-198-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-198-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-199-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-199-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-2-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-2-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-20-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-20-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-200-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-200-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-201-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-201-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-202-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-202-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-203-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-203-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-204-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-204-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-205-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-205-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-206-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-206-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-207-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-207-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-208-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-208-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-209-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:44:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-209-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-21-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-21-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-210-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-210-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-211-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-211-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-212-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-212-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-213-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-213-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-214-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-214-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-215-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-215-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-216-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-216-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-217-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-217-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-218-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-218-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-219-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-219-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-22-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-22-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-220-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-220-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-221-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-221-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-222-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-222-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-223-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-223-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-224-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-224-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-225-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-225-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-226-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-226-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-227-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-227-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-228-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-228-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-229-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-229-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-23-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-23-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-230-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-230-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-231-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-231-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-232-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-232-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-233-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-233-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-234-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-234-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-235-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-235-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-236-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-236-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-237-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-237-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-238-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-238-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-239-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-239-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-24-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-24-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-240-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-240-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-241-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-241-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-242-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-242-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-243-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-243-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-244-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-244-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-245-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-245-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-246-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-246-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-247-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-247-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-248-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-248-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-249-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-249-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-25-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-25-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-250-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-250-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-251-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-251-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-252-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-252-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-253-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-253-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-254-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-254-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-255-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-255-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-256-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-256-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-257-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-257-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-258-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-258-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-259-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-259-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-26-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-26-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-260-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:45:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-260-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-261-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-261-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-262-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-262-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-263-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-263-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-264-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-264-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-265-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-265-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-266-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-266-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-267-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-267-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-268-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-268-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-269-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-269-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-27-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-27-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-270-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-270-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-271-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-271-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-272-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-272-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-273-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-273-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-274-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-274-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-275-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-275-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-276-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-276-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-277-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-277-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-278-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-278-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-279-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-279-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-28-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-28-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-280-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-280-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-281-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-281-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-282-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-282-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-283-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-283-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-284-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-284-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-285-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-285-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-286-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-286-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-287-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-287-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-288-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-288-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-289-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-289-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-29-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-29-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-290-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-290-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-291-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-291-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-292-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-292-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-293-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-293-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-294-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-294-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-295-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-295-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-296-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-296-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-297-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-297-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-298-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-298-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-299-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-299-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-3-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-3-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-30-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-30-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-300-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-300-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-301-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-301-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-302-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-302-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-303-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-303-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-304-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-304-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-305-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-305-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-306-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-306-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-307-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-307-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-308-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-308-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-309-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-309-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-31-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-31-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-310-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:46:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-310-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-311-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-311-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-312-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-312-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-313-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-313-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-314-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-314-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-315-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-315-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-316-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-316-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-317-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-317-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-318-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-318-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-319-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-319-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-32-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-32-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-320-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-320-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-321-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-321-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-322-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-322-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-323-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-323-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-324-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-324-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-325-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-325-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-326-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-326-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-327-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-327-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-328-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-328-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-329-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-329-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-33-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-33-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-330-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-330-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-331-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-331-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-332-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-332-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-333-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-333-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-334-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-334-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-335-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-335-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-336-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-336-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-337-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-337-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-338-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-338-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-339-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-339-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-34-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-34-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-340-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-340-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-341-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-341-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-342-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-342-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-343-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-343-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-344-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-344-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-345-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-345-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-346-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-346-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-347-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-347-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-348-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-348-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-349-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-349-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-35-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-35-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-350-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-350-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-351-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-351-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-352-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-352-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-353-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-353-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-354-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-354-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-355-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-355-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-356-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-356-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-357-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-357-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-358-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-358-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-359-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-359-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-36-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:47:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-36-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-360-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-360-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-361-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-361-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-362-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-362-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-363-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-363-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-364-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-365-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-365-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-366-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-366-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-367-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-368-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-369-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-37-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-37-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-370-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-371-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-372-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-373-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-374-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-375-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-376-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-377-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-378-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-379-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-38-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-38-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-380-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-381-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-382-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-383-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-384-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-385-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-386-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-387-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-388-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-389-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-39-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-39-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-390-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-391-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-392-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-393-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-394-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-395-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-396-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-397-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-398-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-399-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-4-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-4-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-40-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-40-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-400-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-401-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-402-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-403-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-404-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-405-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-406-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-407-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-408-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-409-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:56 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-41-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-41-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-410-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:48:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-411-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-412-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-413-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-414-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-415-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-416-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-417-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-418-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-419-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-42-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-42-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-420-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-421-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-422-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-423-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-424-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-425-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-426-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-427-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-428-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-429-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-43-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-43-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-430-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-431-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-432-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:25 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-433-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-434-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-435-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:29 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-436-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-437-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-438-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-439-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-44-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-44-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-440-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:35 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-441-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-442-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-443-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-444-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-445-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-446-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-447-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-448-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-449-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:45 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-45-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-45-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-450-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-451-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:49:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:48 - MainThread - test_pvc_multi_snapshot_performance - ERROR  - Cannot delete pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3 : Command '['oc', '-n', 'namespace-pas-test-ff8f6cdb7dc84cc398644', 'delete', 'volumesnapshot', 'pvc-snap-452-4fcfe141d3ed4f8fa47cda7eaaabdc3']' timed out after 600 seconds
>04:59:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-46-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-46-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-47-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-47-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-48-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-48-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-49-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-49-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-5-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-5-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:52 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-50-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-50-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:53 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-51-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-51-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:54 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-52-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-52-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:55 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-53-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-53-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:57 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-54-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-54-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:58 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-55-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-55-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:59 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-56-4fcfe141d3ed4f8fa47cda7eaaabdc3
>04:59:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-56-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:00 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-57-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-57-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:01 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-58-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-58-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:02 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-59-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-59-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:03 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-6-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-6-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:04 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-60-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-60-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:05 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-61-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-61-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:06 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-62-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-62-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:07 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-63-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-63-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:08 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-64-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-64-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:09 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-65-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-65-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:10 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-66-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-66-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:11 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-67-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-67-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:12 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-68-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-68-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:13 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-69-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-69-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:14 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-7-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-7-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:15 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-70-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-70-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:16 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-71-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-71-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:17 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-72-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-72-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:18 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-73-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-73-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:19 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-74-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-74-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:20 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-75-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-75-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:21 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-76-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-76-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-77-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-77-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-78-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-78-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:24 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-79-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-79-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:26 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-8-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-8-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:27 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-80-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-80-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:28 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-81-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-81-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:30 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-82-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-82-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:31 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-83-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-83-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:32 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-84-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-84-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:33 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-85-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-85-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:34 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-86-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-86-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:36 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-87-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-87-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:37 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-88-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-88-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:38 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-89-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-89-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:39 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-9-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-9-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:40 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-90-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-90-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:41 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-91-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-91-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:42 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-92-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-92-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:43 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-93-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-93-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:44 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-94-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-94-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:46 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-95-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-95-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:47 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-96-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-96-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:48 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-97-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-97-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:49 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-98-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-98-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:50 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete pvc-snap-99-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete volumesnapshot pvc-snap-99-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:00:51 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Deleting the test POD : pod-test-rbd-2f1314eacb67426f8ceded15548
>05:00:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete Pod pod-test-rbd-2f1314eacb67426f8ceded15548
>05:01:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Wait until the pod is deleted.
>05:01:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>05:01:22 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): pods "pod-test-rbd-2f1314eacb67426f8ceded15548" not found
>
>05:01:22 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pod-test-rbd-2f1314eacb67426f8ceded15548 of kind: Pod, selector: None, Error: Error during execution of command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml.
>Error is Error from server (NotFound): pods "pod-test-rbd-2f1314eacb67426f8ceded15548" not found
>
>05:01:22 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>05:01:22 - MainThread - ocs_ci.ocs.ocp - INFO  - Pod pod-test-rbd-2f1314eacb67426f8ceded15548 got deleted successfully
>05:01:22 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Delete the PVC : pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:01:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3
>05:01:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Wait until the pvc is deleted.
>05:01:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>05:01:23 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): persistentvolumeclaims "pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3" not found
>
>05:01:23 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 of kind: PersistentVolumeClaim, selector: None, Error: Error during execution of command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml.
>Error is Error from server (NotFound): persistentvolumeclaims "pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3" not found
>
>05:01:23 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>05:01:23 - MainThread - ocs_ci.ocs.ocp - INFO  - PersistentVolumeClaim pvc-test-4fcfe141d3ed4f8fa47cda7eaaabdc3 got deleted successfully
>05:01:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Try to delete the backend PV : pvc-a1eb3923-8867-4771-baec-51eccd76b408
>05:01:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to format output as list
>05:01:23 - MainThread - ocs_ci.helpers.performance_lib - INFO  - Going to run oc --kubeconfig /root/ocpcluster/auth/kubeconfig -n openshift-storage delete pv pvc-a1eb3923-8867-4771-baec-51eccd76b408 with timeout of 600
>05:01:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Deleting the test StorageClass : pas-testing-rbd
>05:01:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc delete StorageClass pas-testing-rbd
>05:01:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Wait until the SC is deleted.
>05:01:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get StorageClass pas-testing-rbd -o yaml
>05:01:23 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): storageclasses.storage.k8s.io "pas-testing-rbd" not found
>
>05:01:23 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pas-testing-rbd of kind: StorageClass, selector: None, Error: Error during execution of command: oc get StorageClass pas-testing-rbd -o yaml.
>Error is Error from server (NotFound): storageclasses.storage.k8s.io "pas-testing-rbd" not found
>
>05:01:23 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>05:01:23 - MainThread - ocs_ci.ocs.ocp - INFO  - StorageClass pas-testing-rbd got deleted successfully
>05:01:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Deleting the test Snapshot Class : pas-test-rbd-snapshot-class
>05:01:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc delete VolumeSnapshotClass pas-test-rbd-snapshot-class
>05:01:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Wait until the VSC is deleted.
>05:01:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get VolumeSnapshotClass pas-test-rbd-snapshot-class -o yaml
>05:01:23 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io "pas-test-rbd-snapshot-class" not found
>
>05:01:23 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pas-test-rbd-snapshot-class of kind: VolumeSnapshotClass, selector: None, Error: Error during execution of command: oc get VolumeSnapshotClass pas-test-rbd-snapshot-class -o yaml.
>Error is Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io "pas-test-rbd-snapshot-class" not found
>
>05:01:23 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>05:01:23 - MainThread - ocs_ci.ocs.ocp - INFO  - VolumeSnapshotClass pas-test-rbd-snapshot-class got deleted successfully
>05:01:23 - MainThread - test_pvc_multi_snapshot_performance - INFO  - Deleting the test storage pool : pas-testing-rbd
>05:01:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage delete CephBlockPool pas-testing-rbd --wait=false
>05:01:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage patch cephblockpool pas-testing-rbd --type=merge -p '{"metadata":{"finalizers":null}}'
>05:01:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool set ocs-storagecluster-cephblockpool target_size_ratio 0.49
>05:01:54 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: set pool 1 target_size_ratio to 0.49
>
>05:01:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool set ocs-storagecluster-cephfilesystem-data0 target_size_ratio 0.49
>05:01:55 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: set pool 10 target_size_ratio to 0.49
>
>05:01:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool ls
>05:01:56 - MainThread - test_pvc_multi_snapshot_performance - WARNING  - The pool did not deleted by CSI, forcing delete it manually
>05:01:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage rsh rook-ceph-tools-749dfcffb9-jr87p ceph osd pool delete pas-testing-rbd pas-testing-rbd --yes-i-really-really-mean-it
>05:01:56 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: pool 'pas-testing-rbd' removed
>
>05:01:56 - MainThread - ocs_ci.ocs.perftests - INFO  - Deleting the test namespace : namespace-pas-test-ff8f6cdb7dc84cc398644
>05:01:56 - MainThread - ocs_ci.ocs.ocp - INFO  - Switching to project openshift-storage
>05:01:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc project openshift-storage
>05:01:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 delete Project namespace-pas-test-ff8f6cdb7dc84cc398644
>05:11:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage delete Secret secret-test-rbd-1cc0e6579c1f4b038d574c3a
>05:11:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Secret secret-test-rbd-1cc0e6579c1f4b038d574c3a -n openshift-storage -o yaml
>05:11:57 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): secrets "secret-test-rbd-1cc0e6579c1f4b038d574c3a" not found
>
>05:11:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: secret-test-rbd-1cc0e6579c1f4b038d574c3a of kind: Secret, selector: None, Error: Error during execution of command: oc -n openshift-storage get Secret secret-test-rbd-1cc0e6579c1f4b038d574c3a -n openshift-storage -o yaml.
>Error is Error from server (NotFound): secrets "secret-test-rbd-1cc0e6579c1f4b038d574c3a" not found
>
>05:11:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>05:11:57 - MainThread - ocs_ci.ocs.ocp - INFO  - Secret secret-test-rbd-1cc0e6579c1f4b038d574c3a got deleted successfully
>05:11:57 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Attempt to remove resource: pod-test-rbd-2f1314eacb67426f8ceded15548 which isalready deleted! Skipping delete of this resource!
>05:11:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml
>05:11:57 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): pods "pod-test-rbd-2f1314eacb67426f8ceded15548" not found
>
>05:11:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pod-test-rbd-2f1314eacb67426f8ceded15548 of kind: Pod, selector: None, Error: Error during execution of command: oc -n namespace-pas-test-ff8f6cdb7dc84cc398644 get Pod pod-test-rbd-2f1314eacb67426f8ceded15548 -n namespace-pas-test-ff8f6cdb7dc84cc398644 -o yaml.
>Error is Error from server (NotFound): pods "pod-test-rbd-2f1314eacb67426f8ceded15548" not found
>
>05:11:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>05:11:57 - MainThread - ocs_ci.ocs.ocp - INFO  - Pod pod-test-rbd-2f1314eacb67426f8ceded15548 got deleted successfully
>05:11:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc wait --for condition=ready pod -l app=rook-ceph-tools -n openshift-storage --timeout=300s
>05:11:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pod -l 'app=rook-ceph-tools' -o jsonpath='{.items[0].metadata.name}'
>05:11:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage exec rook-ceph-tools-749dfcffb9-jr87p -- ceph health
>05:11:58 - MainThread - ocs_ci.utility.utils - INFO  - Ceph cluster health is HEALTH_OK.
>05:11:58 - MainThread - tests2.conftest - INFO  - Ceph health check passed at teardown
>05:11:58 - Dummy-1 - ocs_ci.utility.utils - INFO  - Executing command: oc get PersistentVolume  -A -o yaml
>05:11:58 - Dummy-4 - ocs_ci.utility.utils - INFO  - Executing command: oc get StorageClass  -A -o yaml
>05:11:58 - Dummy-8 - ocs_ci.utility.utils - INFO  - Executing command: oc get PersistentVolumeClaim  -A -o yaml
>05:11:58 - Dummy-5 - ocs_ci.utility.utils - INFO  - Executing command: oc get CephFileSystem  -A -o yaml
>05:11:58 - Dummy-9 - ocs_ci.utility.utils - INFO  - Executing command: oc get Namespace  -A -o yaml
>05:11:58 - Dummy-10 - ocs_ci.utility.utils - INFO  - Executing command: oc get CephBlockPool  -A -o yaml
>05:11:58 - Dummy-11 - ocs_ci.utility.utils - INFO  - Executing command: oc get volumesnapshot  -A -o yaml
>05:11:58 - Dummy-6 - ocs_ci.utility.utils - INFO  - Executing command: oc get Pod  -A -o yaml
>
>tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py::TestPvcMultiSnapshotPerformance::test_pvc_multiple_snapshot_performance[CephBlockPool-512] ERROR
>
>==================================== ERRORS ====================================
>_ ERROR at teardown of TestPvcMultiSnapshotPerformance.test_pvc_multiple_snapshot_performance[CephBlockPool-512] _
>
>self = <test_pvc_multi_snapshot_performance.TestPvcMultiSnapshotPerformance object at 0x7f96605d4fd0>
>
>    def teardown(self):
>        """
>        Cleaning up the environment :
>            Delete all snapshot
>            Delete the POD
>            Delete the PVC and the PV
>            Delete the StorageClass
>            Delete the VolumeSnapshotClass
>            Delete the data pool
>            Switch to the default namespace
>            Delete the tested namespace
>    
>        """
>        log.info("Cleanup the test environment")
>    
>        if self.full_teardown:
>            # Getting the name of the PCV's backed PV
>            try:
>                pv = self.pvc_obj.get("spec")["spec"]["volumeName"]
>            except KeyError:
>                log.error(
>                    f"Cannot found key in the PVC object {json.dumps(self.pvc_obj.get('spec').get('spec'), indent=3)}"
>                )
>    
>            # Getting the list of all snapshots
>            try:
>                snapshot_list = self.snapshot.get(all_namespaces=True)["items"]
>            except Exception as err:
>                log.error(f"Cannot get the list of snapshots : {err}")
>                snapshot_list = []
>    
>            # Deleting al snapshots from the cluster
>            log.info(f"Trying to delete all ({len(snapshot_list)}) Snapshots")
>            log.debug(
>                f"The list of all snapshots is : {json.dumps(snapshot_list, indent=3)}"
>            )
>            for vs in snapshot_list:
>                snap_name = vs["metadata"]["name"]
>                log.info(f"Try to delete {snap_name}")
>                try:
>                    self.snapshot.delete(resource_name=snap_name)
>                except Exception as err:
>                    log.error(f"Cannot delete {snap_name} : {err}")
>    
>            # Deleting the pod which wrote data to the pvc
>            log.info(f"Deleting the test POD : {self.pod_obj.name}")
>            try:
>                self.pod_obj.delete()
>                log.info("Wait until the pod is deleted.")
>                self.pod_obj.ocp.wait_for_delete(resource_name=self.pod_obj.name)
>            except Exception as ex:
>                log.error(f"Cannot delete the test pod : {ex}")
>    
>            # Deleting the PVC which used in the test.
>            try:
>                log.info(f"Delete the PVC : {self.pvc_obj.name}")
>                self.pvc_obj.delete()
>                log.info("Wait until the pvc is deleted.")
>                self.pvc_obj.ocp.wait_for_delete(resource_name=self.pvc_obj.name)
>            except Exception as ex:
>                log.error(f"Cannot delete the test pvc : {ex}")
>    
>            # Delete the backend PV of the PVC
>            log.info(f"Try to delete the backend PV : {pv}")
>            try:
>                run_oc_command(f"delete pv {pv}")
>            except Exception as ex:
>                err_msg = f"cannot delete PV {pv} - [{ex}]"
>                log.error(err_msg)
>    
>            # Deleting the StorageClass used in the test
>            log.info(f"Deleting the test StorageClass : {self.sc_obj.name}")
>            try:
>                self.sc_obj.delete()
>                log.info("Wait until the SC is deleted.")
>                self.sc_obj.ocp.wait_for_delete(resource_name=self.sc_obj.name)
>            except Exception as ex:
>                log.error(f"Can not delete the test sc : {ex}")
>    
>            # Deleting the VolumeSnapshotClass used in the test
>            log.info(f"Deleting the test Snapshot Class : {self.snap_class.name}")
>            try:
>                self.snap_class.delete()
>                log.info("Wait until the VSC is deleted.")
>                self.snap_class.ocp.wait_for_delete(resource_name=self.snap_class.name)
>            except Exception as ex:
>                log.error(f"Can not delete the test vsc : {ex}")
>    
>            # Deleting the Data pool
>            log.info(f"Deleting the test storage pool : {self.sc_name}")
>            self.delete_ceph_pool(self.sc_name)
>            # Verify deletion by checking the backend CEPH pools using the toolbox
>            results = self.ceph_cluster.toolbox.exec_cmd_on_pod("ceph osd pool ls")
>            log.debug(f"Existing pools are : {results}")
>            if self.sc_name in results.split():
>                log.warning(
>                    "The pool did not deleted by CSI, forcing delete it manually"
>                )
>                self.ceph_cluster.toolbox.exec_cmd_on_pod(
>                    f"ceph osd pool delete {self.sc_name} {self.sc_name} "
>                    "--yes-i-really-really-mean-it"
>                )
>            else:
>                log.info(f"The pool {self.sc_name} was deleted successfully")
>    
>        # Deleting the namespace used by the test
>>       self.delete_test_project()
>
>tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py:188: 
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>ocs_ci/ocs/perftests.py:886: in delete_test_project
>    self.proj.delete(resource_name=self.namespace)
>ocs_ci/ocs/ocp.py:379: in delete
>    return self.exec_oc_cmd(command)
>ocs_ci/ocs/ocp.py:163: in exec_oc_cmd
>    out = run_cmd(
>ocs_ci/utility/utils.py:473: in run_cmd
>    completed_process = exec_cmd(
>ocs_ci/utility/utils.py:606: in exec_cmd
>    completed_process = subprocess.run(
>/usr/lib64/python3.8/subprocess.py:495: in run
>    stdout, stderr = process.communicate(input, timeout=timeout)
>/usr/lib64/python3.8/subprocess.py:1028: in communicate
>    stdout, stderr = self._communicate(input, endtime, timeout)
>/usr/lib64/python3.8/subprocess.py:1869: in _communicate
>    self._check_timeout(endtime, orig_timeout, stdout, stderr)
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>
>self = <subprocess.Popen object at 0x7f9658fb1520>, endtime = 18440.769447703
>orig_timeout = 600
>stdout_seq = [b'project.project.openshift.io "namespace-pas-test-ff8f6cdb7dc84cc398644" deleted\n']
>stderr_seq = [b'Warning: deleting cluster-scoped resources, not scoped to the provided namespace\n']
>skip_check_and_raise = False
>
>    def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq,
>                       skip_check_and_raise=False):
>        """Convenience for checking if a timeout has expired."""
>        if endtime is None:
>            return
>        if skip_check_and_raise or _time() > endtime:
>>           raise TimeoutExpired(
>                    self.args, orig_timeout,
>                    output=b''.join(stdout_seq) if stdout_seq else None,
>                    stderr=b''.join(stderr_seq) if stderr_seq else None)
>E           subprocess.TimeoutExpired: Command '['oc', '-n', 'namespace-pas-test-ff8f6cdb7dc84cc398644', 'delete', 'Project', 'namespace-pas-test-ff8f6cdb7dc84cc398644']' timed out after 600 seconds
>
>/usr/lib64/python3.8/subprocess.py:1072: TimeoutExpired
>=================================== FAILURES ===================================
>_ TestPvcMultiSnapshotPerformance.test_pvc_multiple_snapshot_performance[CephBlockPool-512] _
>
>self = <test_pvc_multi_snapshot_performance.TestPvcMultiSnapshotPerformance object at 0x7f96605d4fd0>
>pvc_factory = <function pvc_factory_fixture.<locals>.factory at 0x7f9667a95af0>
>pod_factory = <function pod_factory_fixture.<locals>.factory at 0x7f9667a95ca0>
>secret_factory = <function secret_factory_fixture.<locals>.factory at 0x7f9667a95dc0>
>interface_type = 'CephBlockPool', snap_number = 512
>
>    @pytest.mark.polarion_id("OCS-2623")
>    @pytest.mark.parametrize(
>        argnames=["interface_type", "snap_number"],
>        argvalues=[
>            pytest.param(*[constants.CEPHBLOCKPOOL, 512]),
>            pytest.param(*[constants.CEPHFILESYSTEM, 100]),
>        ],
>    )
>    def test_pvc_multiple_snapshot_performance(
>        self,
>        pvc_factory,
>        pod_factory,
>        secret_factory,
>        interface_type,
>        snap_number,
>    ):
>        """
>        1. Creating PVC
>           size is depend on storage capacity, but not less then 1 GiB
>           it will use ~75% capacity of the Storage, Min storage capacity 1 TiB
>        2. Fill the PVC with 80% of data
>        3. Take a snapshot of the PVC and measure the total and CSI times of creation.
>        4. re-write the data on the PVC
>        5. Take a snapshot of the PVC and measure the total and the CSI times of creation.
>        6. repeat steps 4-5 the numbers of snapshot we want to take : 512
>           this will be run by outside script for low memory consumption
>        7. print all information.
>    
>        Raises:
>            StorageNotSufficientException: in case of not enough capacity
>    
>        """
>    
>        # Getting the full path for the test logs
>        self.results_path = get_full_test_logs_path(cname=self)
>        self.full_log_path = f"{self.results_path}-{interface_type}-{snap_number}"
>        log.info(f"Logs file path name is : {self.full_log_path}")
>        log.info(f"Reslut path is : {self.results_path}")
>    
>        self.full_teardown = True
>        self.num_of_snaps = snap_number
>        if self.dev_mode:
>            self.num_of_snaps = 2
>    
>        log.info(f"Going to create {self.num_of_snaps} {interface_type} snapshots")
>    
>        # since we do not want to use more then 65%, we add 35% to the needed
>        # capacity, and minimum PVC size is 1 GiB
>        self.need_capacity = int((self.num_of_snaps + 2) * 1.35)
>    
>        # Test will run only on system with enough capacity
>        if self.capacity_to_use < self.need_capacity:
>            err_msg = (
>                f"The system have only {self.ceph_capacity} GiB, "
>                f"we want to use only {self.capacity_to_use} GiB, "
>                f"and we need {self.need_capacity} GiB to run the test"
>            )
>            log.error(err_msg)
>            raise exceptions.StorageNotSufficientException(err_msg)
>    
>        # Calculating the PVC size in GiB
>        self.pvc_size = int(self.capacity_to_use / (self.num_of_snaps + 2))
>        if self.dev_mode:
>            self.pvc_size = 5
>    
>        self.interface = interface_type
>        self.sc_name = "pas-testing-rbd"
>        pool_name = self.sc_name
>        if self.interface == constants.CEPHFILESYSTEM:
>            self.sc_name = "pas-testing-cephfs"
>            pool_name = f"{self.sc_name}-data0"
>    
>        # Creating new storage pool
>        self.create_new_pool(self.sc_name)
>    
>        # Creating new StorageClass (pool) for the test.
>        secret = secret_factory(interface=self.interface)
>        self.sc_obj = helpers.create_storage_class(
>            interface_type=self.interface,
>            interface_name=pool_name,
>            secret_name=secret.name,
>            sc_name=self.sc_name,
>            fs_name=self.sc_name,
>        )
>        log.info(f"The new SC is : {self.sc_obj.name}")
>        log.debug(f"All SC data is {json.dumps(self.sc_obj.data, indent=3)}")
>    
>        # Create new VolumeSnapshotClass
>        self.snap_class = self.create_snapshotclass(self.interface)
>    
>        # Create new PVC
>        log.info(f"Creating {self.pvc_size} GiB PVC of {interface_type}")
>        self.pvc_obj = pvc_factory(
>            interface=self.interface,
>            storageclass=self.sc_obj,
>            size=self.pvc_size,
>            status=constants.STATUS_BOUND,
>            project=self.proj,
>        )
>    
>        # Create POD which will attache to the new PVC
>        log.info("Creating A POD")
>        self.pod_obj = pod_factory(
>            interface=self.interface,
>            pvc=self.pvc_obj,
>            status=constants.STATUS_RUNNING,
>            pod_dict_path=constants.PERF_POD_YAML,
>        )
>    
>        # Calculating the file size as 80% of the PVC size
>        self.filesize = self.pvc_obj.size * 0.80
>        # Change the file size to MB for the FIO function
>        self.file_size = f"{int(self.filesize * constants.GB2MB)}M"
>        self.file_name = self.pod_obj.name
>    
>        log.info(
>            f"Total capacity size is : {self.ceph_capacity} GiB, "
>            f"Going to use {self.need_capacity} GiB, "
>            f"With {self.num_of_snaps} Snapshots to {self.pvc_size} GiB PVC. "
>            f"File size to be written is : {self.file_size} "
>            f"with the name of {self.file_name}"
>        )
>    
>        # Reading basic snapshot yaml file
>        self.snap_yaml = constants.CSI_CEPHFS_SNAPSHOT_YAML
>        self.sc = constants.DEFAULT_VOLUMESNAPSHOTCLASS_CEPHFS
>        self.fs_type = "cephfs"
>        if interface_type == constants.CEPHBLOCKPOOL:
>            self.snap_yaml = constants.CSI_RBD_SNAPSHOT_YAML
>            self.fs_type = "rbd"
>            self.sc = constants.DEFAULT_VOLUMESNAPSHOTCLASS_RBD
>        with open(self.snap_yaml, "r") as stream:
>            try:
>                self.snap_templ = yaml.safe_load(stream)
>                self.snap_templ["spec"]["volumeSnapshotClassName"] = self.sc
>                self.snap_templ["spec"]["source"][
>                    "persistentVolumeClaimName"
>                ] = self.pvc_obj.name
>            except yaml.YAMLError as exc:
>                log.error(f"Can not read template yaml file {exc}")
>        log.debug(
>            f"Snapshot yaml file : {self.snap_yaml} "
>            f"Content of snapshot yaml file {json.dumps(self.snap_templ, indent=4)}"
>        )
>    
>        self.build_fio_command()
>        self.start_time = self.get_time()
>    
>        # Initialize the results doc file.
>        full_results = self.init_full_results(
>            ResultsAnalyse(
>                self.uuid, self.crd_data, self.full_log_path, "multiple_snapshots"
>            )
>        )
>>       full_results.all_results = self.run()
>
>tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py:556: 
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py:384: in run
>    ct, sci_ct = self.create_snapshot(test_num)
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>
>self = <test_pvc_multi_snapshot_performance.TestPvcMultiSnapshotPerformance object at 0x7f96605d4fd0>
>snap_num = 452
>
>    def create_snapshot(self, snap_num):
>        """
>        Creating snapshot of volume
>        measure the total snapshot creation time
>        and the CSI creation time
>    
>        Args:
>            snap_num (int) the number of snapshot to create
>    
>        Returns:
>            int: the creation time of the snapshot (in sec.)
>    
>        """
>        log.info(f"Taking snapshot number {snap_num}")
>        # Getting UTC time before test starting for log retrieve
>        start_time = self.get_time("csi")
>        snap_name = f"pvc-snap-{snap_num}-"
>        snap_name += self.pvc_obj.name.split("-")[-1]
>        self.snap_templ["metadata"]["name"] = snap_name
>        self.snap_templ["spec"]["volumeSnapshotClassName"] = self.snap_class.name
>    
>        fd, tmpfile = tempfile.mkstemp(suffix=".yaml", prefix="Snap")
>        log.debug(f"Going to create {tmpfile}")
>        with open(tmpfile, "w") as f:
>            yaml.dump(self.snap_templ, f, default_flow_style=False)
>    
>        res = run_oc_command(cmd=f"create -f {tmpfile}", namespace=self.namespace)
>        if ERRMSG in res[0]:
>            err_msg = f"Failed to create snapshot : {res}"
>            log.error(err_msg)
>            raise Exception(err_msg)
>    
>        # wait until snapshot is ready
>        timeout = 720
>        sleep_time = 10
>        snap_con_name = None
>        snap_uid = None
>        while timeout > 0:
>            res = run_oc_command(
>                f"get volumesnapshot {snap_name} -o yaml", namespace=self.namespace
>            )
>    
>            if ERRMSG not in res[0]:
>                res = yaml.safe_load("\n".join(res))
>                log.debug(f"The command output is : {yaml.dump(res)}")
>                try:
>                    if res["status"]["readyToUse"]:
>                        log.info(f"{snap_name} Created and ready to use")
>                        snap_con_name = res["status"]["boundVolumeSnapshotContentName"]
>                        snap_uid = res["metadata"]["uid"]
>                        break
>                    else:
>                        log.info(
>                            f"{snap_name} is not ready yet, sleep {sleep_time} sec before re-check"
>                        )
>                        time.sleep(sleep_time)
>                        timeout -= sleep_time
>                except Exception:
>                    log.info(
>                        f"{snap_name} is not ready yet, sleep {sleep_time} sec before re-check"
>                    )
>                    time.sleep(sleep_time)
>                    timeout -= sleep_time
>    
>            else:
>                err_msg = f"Can not get snapshot status {res}"
>                log.error(err_msg)
>                raise Exception(err_msg)
>        if snap_con_name:
>            creation_time = performance_lib.measure_total_snapshot_creation_time(
>                snap_name, start_time
>            )
>            csi_creation_time = performance_lib.measure_csi_snapshot_creation_time(
>                self.interface, snap_uid, start_time
>            )
>            return (creation_time, csi_creation_time)
>        else:
>            err_msg = "Snapshot was not created on time"
>            log.error(err_msg)
>>           raise TimeoutError(err_msg)
>E           TimeoutError: Snapshot was not created on time
>
>tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py:367: TimeoutError
>=========================== short test summary info ============================
>FAILED tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py::TestPvcMultiSnapshotPerformance::test_pvc_multiple_snapshot_performance[CephBlockPool-512]
>ERROR tests2/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py::TestPvcMultiSnapshotPerformance::test_pvc_multiple_snapshot_performance[CephBlockPool-512]
>=================== 1 failed, 1 error in 14610.79s (4:03:30) ===================

Actions: View

Attachments on bug 2149663: 1928758