Hide Forgot
So trying to understand the important items here: 1. It's on VMware LSO - what about other platforms? 2. What's the difference between the Ceph versions? 3. Is it on the same *OCP* versions? 4. RHCOS version? 5. I assume 'append' is the main issue? How's Ceph doing? (ceph status would be nice to see!)
all logs and must-gather can be found at : http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-1879866/(In reply to Yaniv Kaul from comment #2) > So trying to understand the important items here: > 1. It's on VMware LSO - what about other platforms? did not tested, yet > 2. What's the difference between the Ceph versions? Same ceph version > 3. Is it on the same *OCP* versions? yes - it is on the same OCP, only OCS was upgrade > 4. RHCOS version? > 5. I assume 'append' is the main issue? You are referring the smallfile test, while this BZ is for the FIO test one page above in the report. > > How's Ceph doing? (ceph status would be nice to see!) All logs and must-gather can be found at : http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-1879866/
(In reply to Avi Liani from comment #3) > all logs and must-gather can be found at : > http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-1879866/(In > reply to Yaniv Kaul from comment #2) > > So trying to understand the important items here: > > 1. It's on VMware LSO - what about other platforms? > did not tested, yet > > > 2. What's the difference between the Ceph versions? > Same ceph version > > > 3. Is it on the same *OCP* versions? > yes - it is on the same OCP, only OCS was upgrade > > > 4. RHCOS version? # oc get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME compute-0 Ready worker 3d v1.18.3+6c42de8 10.1.160.85 10.1.160.85 Red Hat Enterprise Linux CoreOS 45.82.202008290529-0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.18.3-11.rhaos4.5.gite5bcc71.el8 compute-1 Ready worker 3d v1.18.3+6c42de8 10.1.160.105 10.1.160.105 Red Hat Enterprise Linux CoreOS 45.82.202008290529-0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.18.3-11.rhaos4.5.gite5bcc71.el8 compute-2 Ready worker 3d v1.18.3+6c42de8 10.1.160.141 10.1.160.141 Red Hat Enterprise Linux CoreOS 45.82.202008290529-0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.18.3-11.rhaos4.5.gite5bcc71.el8 control-plane-0 Ready master 3d v1.18.3+6c42de8 10.1.160.88 10.1.160.88 Red Hat Enterprise Linux CoreOS 45.82.202008290529-0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.18.3-11.rhaos4.5.gite5bcc71.el8 control-plane-1 Ready master 3d v1.18.3+6c42de8 10.1.160.86 10.1.160.86 Red Hat Enterprise Linux CoreOS 45.82.202008290529-0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.18.3-11.rhaos4.5.gite5bcc71.el8 control-plane-2 Ready master 3d v1.18.3+6c42de8 10.1.160.146 10.1.160.146 Red Hat Enterprise Linux CoreOS 45.82.202008290529-0 (Ootpa) 4.18.0-193.14.3.el8_2.x86_64 cri-o://1.18.3-11.rhaos4.5.gite5bcc71.el8 > > > 5. I assume 'append' is the main issue? > You are referring the smallfile test, while this BZ is for the FIO test one > page above in the report. > > > > How's Ceph doing? (ceph status would be nice to see!) > > > All logs and must-gather can be found at : > http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-1879866/
(In reply to Avi Liani from comment #3) > all logs and must-gather can be found at : > http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-1879866/(In > reply to Yaniv Kaul from comment #2) > > So trying to understand the important items here: > > 1. It's on VMware LSO - what about other platforms? > did not tested, yet > > > 2. What's the difference between the Ceph versions? > Same ceph version > > > 3. Is it on the same *OCP* versions? > yes - it is on the same OCP, only OCS was upgrade OCS 4.5 -> 4.6 but the Ceph version did not change? > > 4. RHCOS version? > > > 5. I assume 'append' is the main issue? > You are referring the smallfile test, while this BZ is for the FIO test one > page above in the report. > > > > How's Ceph doing? (ceph status would be nice to see!) > > > All logs and must-gather can be found at : > http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-1879866/ How can I access this machine to examine the logs. HTTP is unusable for analyzing logs. If the Ceph version did not change, there's no reason to believe that there is a bug in CephFS (bz1881316). I looked at: https://docs.google.com/document/d/1thRo0AGK2af2ECUGiLOBQ28UQtzWLX-2iYxeSfdRy9s/edit# This appears to be data from a single run. There really should be at least three runs to do a proper analysis. Finally, these are data path I/O where the MDS is minimally involved and the client I/O pattern will be almost the same as RBD. We have done extensive performance testing in the past which affirms this. I suspect this is a transient performance hiccup of some kind.
Assigning to you Patrick to track the info you requested (needinfo). If it is not a bug please close it.
Moving it out from 4.6, please bring it back if there is sufficient data.