Hello pranith, We are currently doing live testing for a CVE update and will be done by tomorrow. I can provide the setup after that. Does that sound okay ? Thanks kasturi
(In reply to RamaKasturi from comment #11) > Hello pranith, > > We are currently doing live testing for a CVE update and will be done > by tomorrow. I can provide the setup after that. Does that sound okay ? > > Thanks > kasturi Sure let us know once you are done. I will leave a needinfo on you until then, to reflect the status. Thanks a lot! Pranith
Hello Pranith, Below is the setup i have created with latest bits of OCP + OCS which you can use the setup to reproduce the issue. Can you please let me know once you are done with the setup so that i can take it back and continue using that hypervisor ? I have created one file & block pvc and attached them to cirros pods. Clearing need info since i have sent the details over email. Thanks kasturi
Xiubo needed help from QE to recreate the issue. We are at the moment waiting for that. Leaving a needinfo on Kasturi for now.
Adding needinfo on ashmitha as she is trying to reproduce the bug. @ashmita, can you please reproduce the bug and once you hit the issue can you handover the setup to pranith ?
Hello pranith, I think the issue was mostly hit by vignesh and ashmitha and i have not hit it so far in my testing. Thanks kasturi
(In reply to RamaKasturi from comment #24) > Hello pranith, > > I think the issue was mostly hit by vignesh and ashmitha and i have not > hit it so far in my testing. > > Thanks > kasturi Okay. I will wait for Vignesh's setup. Leaving needinfo on him.
(In reply to Pranith Kumar K from comment #25) > (In reply to RamaKasturi from comment #24) > > Hello pranith, > > > > I think the issue was mostly hit by vignesh and ashmitha and i have not > > hit it so far in my testing. > > > > Thanks > > kasturi > > Okay. I will wait for Vignesh's setup. Leaving needinfo on him. Vignesh - Ping? Any luck with recreating this issue?
Hello Pranith, QE is running the setup with the recommended configuration and i do not expect slow I/O. @vignesh, is the hypervisor overloaded ? Do we have enough RAM & CPU resources available ? can you please confirm. Thanks kasturi
(In reply to RamaKasturi from comment #28) > Hello Pranith, > > QE is running the setup with the recommended configuration and i do not > expect slow I/O. > > @vignesh, is the hypervisor overloaded ? Do we have enough RAM & CPU > resources available ? can you please confirm. > > Thanks > kasturi Kasturi, I spoke to Shekhar about this issue. We want to know the configuration of the storage so that we can proceed further. Pranith
Hello vignesh, can you provide the information what is requested by pranith here ? @pranith can you please talk to vignesh for the configuration ? Thanks kasturi
(In reply to RamaKasturi from comment #30) > Hello vignesh, > > can you provide the information what is requested by pranith here ? > > @pranith can you please talk to vignesh for the configuration ? Kasturi, I meant the "recommended configuration" you were referring to in earlier comment. So left a needinfo on you instead of Vignesh. Thanks Pranith > > > Thanks > kasturi
Prasanna, Could you let me know how this bz and bz#1717756 are related? Pranith
Pranith, Looks like it got added automatically while I was doing the clone activity, removing it now. Thanks!
Shekhar, Vignesh and I had a meeting to debug the setup and as per the iostat output, the device has very high w_await numbers in the worst case going to ~15 seconds. Now that we have this information, what we probably need to understand is if the hardware and the layers on top(RAID5 etc) is similar to what we suggest to customers? If yes we may have to revisit if the recommendation is correct. I am not sure who in QE can come up with this information. I am keeping a needinfo on Vignesh for now to gather this information. 13:27:18 :( ⚡ grep -Ew "(sde|Device)" 1559891527-iostat.out | column -t Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.02 0.30 0.12 9.43 8.03 1570.80 330.73 2.95 307.67 1197.21 296.53 101.58 96.98 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 9.00 0.00 1401.10 311.36 79.04 15275.56 0.00 15275.56 111.11 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 7.20 0.00 1251.20 347.56 82.76 8594.61 0.00 8594.61 138.92 100.02 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.20 0.00 5.80 0.00 1434.40 494.62 59.58 10919.28 0.00 10919.28 172.41 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.20 0.00 11.00 0.00 1788.90 325.25 44.88 11414.49 0.00 11414.49 90.91 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.20 0.00 6.20 0.00 1276.80 411.87 56.05 2982.13 0.00 2982.13 161.29 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 8.20 0.00 1519.20 370.54 40.61 7741.29 0.00 7741.29 121.95 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 21.40 0.00 3752.00 350.65 22.52 1936.58 0.00 1936.58 46.73 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 7.80 0.00 1273.60 326.56 36.32 2614.44 0.00 2614.44 128.21 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 1.00 0.00 6.00 0.00 1401.60 467.20 91.34 4979.63 0.00 4979.63 166.67 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 5.80 0.00 1292.80 445.79 86.66 8408.83 0.00 8408.83 172.41 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 5.60 0.00 1383.20 494.00 61.27 13228.00 0.00 13228.00 178.57 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.60 0.00 25.20 0.00 4014.10 318.58 20.38 4842.25 0.00 4842.25 39.68 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 8.20 0.00 1448.00 353.17 30.03 2707.17 0.00 2707.17 121.95 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 8.00 0.00 1491.20 372.80 61.15 2937.35 0.00 2937.35 125.00 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.80 0.00 3.60 0.00 921.60 512.00 54.32 7677.50 0.00 7677.50 277.78 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 12.60 0.00 1858.80 295.05 46.20 8562.62 0.00 8562.62 79.37 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 7.00 0.00 1426.40 407.54 26.85 4439.94 0.00 4439.94 142.86 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.20 0.00 12.60 0.00 2080.80 330.29 25.69 2561.33 0.00 2561.33 79.37 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 6.60 0.00 1069.60 324.12 29.52 3425.52 0.00 3425.52 151.52 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.80 0.00 27.80 0.00 4990.40 359.02 32.25 826.99 0.00 826.99 35.96 99.96 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.40 0.00 3.60 0.00 921.60 512.00 74.01 5144.78 0.00 5144.78 277.83 100.02 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sde 0.00 0.00 0.00 5.40 0.00 1226.40 454.22 90.74 10191.89 0.00 10191.89 185.19 100.00 Pranith
(In reply to RamaKasturi from comment #28) > Hello Pranith, > > QE is running the setup with the recommended configuration and i do not > expect slow I/O. > > @vignesh, is the hypervisor overloaded ? Do we have enough RAM & CPU > resources available ? can you please confirm. > > Thanks > kasturi Vignesh, Kasturi I see Bricks are being created on top of RAID-5. As its a Replica 3 volume, I wanted to understand why are we using RAID-5 at the backend? Is this configuration recommended/documented somewhere? RAID introduces write penalty which further slows the write performance. 60 secs for a write IO is very slow and RAID in backend alone is not a factor but it's a contributing factor. Me, Pranith and Vignesh did a troubleshooting session which points to hardware issue in the environment as it's seen in his comment 36 above.
Pranith, next step? From comments, it doesn't look like there's a bug identified but latency caused due to underlying hardware? Close bz?
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days