Description of problem: ======================= With eager-lock enabled, the slave has more INODELK than when eager-lock is disabled. Given that it is only through geo-rep the slave is being accessed, eager-locking should be more effective, however this doesn't seem to be the case as of now. When eager-lock is disabled, there are minimal number of INODELK's Version-Release number of selected component (if applicable): ============================================================= glusterfs-geo-replication-3.8.4-54.13.el7rhgs.x86_64 How reproducible: ================ Always Steps to Reproduce: =================== Make sure shd has been disabled: 1.Create a 1x3 master and slave volume 2.Enable profile info on both master and slave 3.Start the geo-rep session 4.Make sure eager-lock is enabled on both the master and slave 5.Clear the profile info on master and slave 6.Create a 1 GB file on the master using: dd if=/dev/urandom of=file1.txt bs=1024 count=1000000 7.Take the profile info and save the info for future comparison 8.Clear the profile info on both master and slave 9.Start the geo-rep session 10.Wait for sync and take the profile info of both volumes rerun above test with eager-lock off on both master/slave to compare results Actual results: =============== Excess INODELK seen on the slave with eager-lock enabled Expected results: ================= INODELK on the slave should be minimal
Created attachment 1456438 [details] Comparison when eager-lock is enabled and disabled for the scenarios explained
Analysis: Geo-replication uses rsync to sync data from master volume to slave volume. There will be one mount point per brick. If master is 1*3 and slave is 1*3, master has three master volume mount points and slave has three slave volume mount points mounted locally. But only one mountpoint is used to sync in this case and others will not be used (One mountpoint is used per replica subvolume set). From attachment 2 [details], when eager lock is enabled on slave and during geo-rep syncing a single file to slave, the number of inodelks taken are in thousands where it should be minimal. We observed the similar behaviour without geo-replication. While the file is being written, if the same file read, the number of inodelks increased significantly. I assume the same is happening w.r.t geo-replication as rsync reads the file content to calculate checksum and sync delta. But what's more puzzling is when the eager lock is disabled, the geo-replication sync did not see any spike in number of inodelks ? Pranith/Ravi/Ashish/Xavi, Is there any possible explanation ? It is important as we need to decide what's the best configuration for slave when it's a AFR/EC volume.
(In reply to Kotresh HR from comment #4) > Analysis: > > Geo-replication uses rsync to sync data from master volume to slave volume. > There will be one mount point per brick. If master is 1*3 and slave is 1*3, > master has three master volume mount points and slave has three slave volume > mount points mounted locally. But only one mountpoint is used to sync in > this case and others will not be used (One mountpoint is used per replica > subvolume set). > > From attachment 2 [details], when eager lock is enabled on slave and during > geo-rep syncing a single file to slave, the number of inodelks taken are in > thousands where it should be minimal. > > We observed the similar behaviour without geo-replication. While the file is > being written, if the same file read, the number of inodelks increased > significantly. I assume the same is happening w.r.t geo-replication as rsync > reads the file content to calculate checksum and sync delta. Concurrent reads and writes should only cause an increase on the number of INODELK on a disperse volume when reads are made from a different client (different mount point). > > But what's more puzzling is when the eager lock is disabled, the > geo-replication sync did not see any spike in number of inodelks ? Looking at the profile data I see some unexpected things. The first one is that the first profile shows write access to master volume while georep is disabled. Does that correspond to some initial data to perform the test ? The second thing is more interesting: when georep is enabled with eager-lock disabled, I can only see READ requests on the slave, and on master there's no work done (only a bunch of GETXATTR). Shouldn't there be some READs on master and WRITEs on slave to synchronize the data from master to slave ? The same test when eager-lock is enabled shows READs on master and WRITEs on slave. The reason why FINODELK requests are seen is probably because rsync keeps more than one fd open on the file. AFR doesn't use eager-locking if there's more than one open file descriptor on the same file. If the first test would have synchronized data, we should have seen a very similar profile. Can you check again the scenario with eager-lock disabled and make sure that data is actually synchronized ? I guess we'll see very similar profiles in this case. > > Pranith/Ravi/Ashish/Xavi, > > Is there any possible explanation ? It is important as we need to decide > what's the best configuration for slave when it's a AFR/EC volume.
(In reply to Xavi Hernandez from comment #5) > (In reply to Kotresh HR from comment #4) > > Analysis: > > > > Geo-replication uses rsync to sync data from master volume to slave volume. > > There will be one mount point per brick. If master is 1*3 and slave is 1*3, > > master has three master volume mount points and slave has three slave volume > > mount points mounted locally. But only one mountpoint is used to sync in > > this case and others will not be used (One mountpoint is used per replica > > subvolume set). > > > > From attachment 2 [details], when eager lock is enabled on slave and during > > geo-rep syncing a single file to slave, the number of inodelks taken are in > > thousands where it should be minimal. > > > > We observed the similar behaviour without geo-replication. While the file is > > being written, if the same file read, the number of inodelks increased > > significantly. I assume the same is happening w.r.t geo-replication as rsync > > reads the file content to calculate checksum and sync delta. > > Concurrent reads and writes should only cause an increase on the number of > INODELK on a disperse volume when reads are made from a different client > (different mount point). Is it the same behaviour with AFR as well. The tests run are against 1*3. But, I think rsync, uses the same mountpoint to read and write. If it uses same mount point, it should not result in increase the number of inodelk right? > > > > > But what's more puzzling is when the eager lock is disabled, the > > geo-replication sync did not see any spike in number of inodelks ? > > Looking at the profile data I see some unexpected things. The first one is > that the first profile shows write access to master volume while georep is > disabled. Does that correspond to some initial data to perform the test ? > Yes, this is when the data is written to master keeping geo-rep stopped. > The second thing is more interesting: when georep is enabled with eager-lock > disabled, I can only see READ requests on the slave, and on master there's > no work done (only a bunch of GETXATTR). Shouldn't there be some READs on > master and WRITEs on slave to synchronize the data from master to slave ? > Yes, you are right, no reads on master and no writes on slave as well. Needs a re-test. > The same test when eager-lock is enabled shows READs on master and WRITEs on > slave. > > The reason why FINODELK requests are seen is probably because rsync keeps > more than one fd open on the file. AFR doesn't use eager-locking if there's > more than one open file descriptor on the same file. So this means, on geo-rep slave volume, enabling/disabling eager lock has no effect? > > If the first test would have synchronized data, we should have seen a very > similar profile. > > Can you check again the scenario with eager-lock disabled and make sure that > data is actually synchronized ? I guess we'll see very similar profiles in > this case. Yes, I will check with QE to redo the mentioned test. Rallan/Nag, Could you get this testd once more? > > > > > Pranith/Ravi/Ashish/Xavi, > > > > Is there any possible explanation ? It is important as we need to decide > > what's the best configuration for slave when it's a AFR/EC volume.
(In reply to Kotresh HR from comment #6) > (In reply to Xavi Hernandez from comment #5) > > Concurrent reads and writes should only cause an increase on the number of > > INODELK on a disperse volume when reads are made from a different client > > (different mount point). > > Is it the same behaviour with AFR as well. The tests run are against 1*3. > But, I think rsync, uses the same mountpoint to read and write. If it uses > same mount point, it should not result in increase the number of inodelk > right? If it comes from the same client, EC shouldn't show an increase of INODELKs. On AFR, if there are two open fd's, even if they come from the same client, an INODELK increase is expected, but it shouldn't be higher than when eager-lock is disabled (in fact they should be almost equal). > > The reason why FINODELK requests are seen is probably because rsync keeps > > more than one fd open on the file. AFR doesn't use eager-locking if there's > > more than one open file descriptor on the same file. > > So this means, on geo-rep slave volume, enabling/disabling eager lock has no > effect? If I'm right about AFR and rsync really opens more than one fd on the same file, then I fear that eager-lock is useless on slaves. Anyway Pranith has a better understanding of AFR. Let's see if there's something I missed.
(In reply to Xavi Hernandez from comment #7) > If I'm right about AFR and rsync really opens more than one fd on the same > file, > then I fear that eager-lock is useless on slaves. I mean slaves using AFR. In theory with EC this shouldn't happen.
I accidentally deleted some needinfos...
(In reply to Xavi Hernandez from comment #7) > (In reply to Kotresh HR from comment #6) > > (In reply to Xavi Hernandez from comment #5) > > > Concurrent reads and writes should only cause an increase on the number of > > > INODELK on a disperse volume when reads are made from a different client > > > (different mount point). > > > > Is it the same behaviour with AFR as well. The tests run are against 1*3. > > But, I think rsync, uses the same mountpoint to read and write. If it uses > > same mount point, it should not result in increase the number of inodelk > > right? > > If it comes from the same client, EC shouldn't show an increase of INODELKs. > On AFR, if there are two open fd's, even if they come from the same client, > an INODELK increase is expected, but it shouldn't be higher than when > eager-lock is disabled (in fact they should be almost equal). This is correct > > > > The reason why FINODELK requests are seen is probably because rsync keeps > > > more than one fd open on the file. AFR doesn't use eager-locking if there's > > > more than one open file descriptor on the same file. > > > > So this means, on geo-rep slave volume, enabling/disabling eager lock has no > > effect? > > If I'm right about AFR and rsync really opens more than one fd on the same > file, > then I fear that eager-lock is useless on slaves. This is correct. Until AFR moves to the locks infra added by Xavi for EC, we will have to live with this. > > Anyway Pranith has a better understanding of AFR. Let's see if there's > something I missed.