Bug 830606

Something else as well.

While backups are running over the NFS mounts I am getting hangs in various things such as Chrome and even doing a ps listing.

ps will list but then will not return to the command prompt.

Chrome keeps putting up messages about page still loading and eventually tabs will appear to totally hang.

.

And this machine has an SSD drive if that makes any difference.

.

Is this still happening with 3.5.3 kernels in updates?

I won't be at the location where the NAS box is located for a few weeks so I cannot test this right now.

I'm still running 3.3.7 currently.

I'll make a note to upgrade before I go to the NAS box location and test this again.

.

Note there's a known problem with stat'ing while writing: in order to get the (posix-required) up-to-date file times from the server, the client needs to first make sure any dirty data's written out to the server.

If that's the problem I think tuning the vm to limit the number of dirty pages might help (e.g., dirty_bytes or dirty_ratio--see Documentation/sysctl/vm.txt in the kernel source tree.)

As I said, I'm using an SSD in this machine and I have it mounted like this:
defaults,noatime,discard

in order to reduce unnecessary writes to the drive.

Is the noatime affecting NFS?

.

Take the "discard" mount option out on the local file system. It is off by default since it can (depending on the device) be very slow.

I really do not want to do that.

Without 'discard' the SSD will eventually get slower and slower.  There are multiple sites that have tested SSD performance and proven this.

I got discard working and it tests just fine as it zeroes out all the sectors on files that have been removed.

I'm using a SanDisk Extreme SSD and have not noticed any slowness at all.  In fact quite the opposite.  And it's stayed just as responsive as the day I installed it.

.

Could you test without discard and then report the results?

Yes, I can disable discard temporarily to run a test.

As I said, it'll be few weeks before I'm back where I can test it.

.

(In reply to comment #10)
> Yes, I can disable discard temporarily to run a test.
> 
> As I said, it'll be few weeks before I'm back where I can test it.
> 

Thanks. It is important to rule out discard because as Ric mentioned, depending on the device, the -o discard mount option could make things very slow. Especially in workloads with lots of freed blocks and sync, which might not be your case, but it's worth testing anyway.

Note that if this happens to be the cause of the slowness you can always use batched discard (see fstrim(8) from util-linux) letting it run once in a while (daily, weekly, monthly) and get rid of the -o discard option.

Thanks!
-Lukas

I'm a little fuzzy on the actual configuration here. The original description said:


Steps to Reproduce:
1. Install F17
2. Create NFS mount for NAS box
3. Start backups.

What sort of server is this NAS box? When you said that the machine had a SSD, you meant the NFS client, right? If so, then it's not clear to me that messing around with mount options involving it will have any effect on NFS performance...

Good question - I had assumed that the problem was exporting via NFS the SSD file system.

No.  It's the other way around.

The client laptop has the SSD drive.

The NAS box has regular HDD drives.

.

Ok, thanks. So the problem is slow "ls" performance while running heavy I/O (mostly writes?) to the NAS. You also said:

   "On other non-F17 machines this works without any problem."

What other non-F17 machines seem to work better here?

Also:

How much RAM is in the client?

Does it work better if you use an unaliased ls command with no options? e.g.:

    $ /bin/ls /mnt/Backup_Filesets_?

I have been using the same setup for NFS mounting the NAS box since F11 and everything worked normally until I hit F17.  F16, F15 were fine.

RAM:
# cat /proc/meminfo | head -1
MemTotal:        8075568 kB

ls is aliased as 'ls --color=auto'.

Have not seen any other problems at all with ls.

And it's not just ls.  It's Chrome weirdness, ps listings not returning.

.

(In reply to comment #16)
> I have been using the same setup for NFS mounting the NAS box since F11 and
> everything worked normally until I hit F17.  F16, F15 were fine.
> 

Ok, so this is a regression -- good to know. Is there some reason you haven't updated to 3.5.z?


> RAM:
> # cat /proc/meminfo | head -1
> MemTotal:        8075568 kB
> 

Ok, so quite a bit of RAM. Bruce's theory is quite plausible in this case.


> ls is aliased as 'ls --color=auto'.
> 
> Have not seen any other problems at all with ls.
> 
> And it's not just ls.  It's Chrome weirdness, ps listings not returning.
> 
> .

Yep, that's one of the things you'll need to separate out. "ls --color=auto" means that ls has to first do a readdir() (or similar) to determine what files are in a directory and then stat() each one in order to colorize them properly.

Determining which of those operations is taking the bulk of the time would be helpful. Most likely, it's one of the stat() operations since, as Bruce points out, POSIX requires that any cached data be flushed to the server prior to returning the results of that operation. When a file continually being written to, then that can give you a livelock sort of situation...

I have upgraded to 3.5.1 but have just not had the opportunity to get back to the NAS box location to test it yet.

I understand the ls situation but even without ls lots of things hanging.

I'll report back once I can test again.

.

I think I may have the same problem. When I uncompress a file on my nfs mount, iotop shows 80KB/s max write rate, 50% to 100% in the IO> column (iowait?) on the client side.

20715 be/4 edouard     0.00 B/s   70.99 K/s  0.00 % 17.83 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s   50.97 K/s  0.00 % 39.00 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s   58.78 K/s  0.00 % 37.35 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s   19.61 K/s  0.00 % 37.81 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s  117.63 K/s  0.00 % 36.78 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s   70.58 K/s  0.00 % 30.27 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s  176.51 K/s  0.00 % 38.20 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s   39.22 K/s  0.00 % 28.67 % tar zxf linux-2.6.32.9-apc.tar.gz
20715 be/4 edouard     0.00 B/s   31.37 K/s  0.00 % 28.69 % tar zxf linux-2.6.32.9-apc.tar.gz

On the server side, some nfsd take some io but I have 100% IOwait on jbd2/md2-8

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s    0.00 B/s  0.00 % 58.05 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s  276.69 K/s  0.00 %  0.00 % [nfsd]
Total DISK READ: 5.52 M/s | Total DISK WRITE: 499.34 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s    3.87 K/s  0.00 % 99.99 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s   23.23 K/s  0.00 % 13.14 % [nfsd]
  483 be/3 root        0.00 B/s    7.74 K/s  0.00 %  5.02 % [jbd2/dm-0-8]
Total DISK READ: 6.12 M/s | Total DISK WRITE: 464.52 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s    3.87 K/s  0.00 % 99.99 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s   30.97 K/s  0.00 % 14.22 % [nfsd]
Total DISK READ: 5.23 M/s | Total DISK WRITE: 537.80 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s   33.97 K/s  0.00 % 97.43 % [jbd2/md2-8]
 1708 be/4 root        0.00 B/s   86.80 K/s  0.00 % 21.05 % [nfsd]
Total DISK READ: 6.71 M/s | Total DISK WRITE: 576.86 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s   38.72 K/s  0.00 % 96.89 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s  104.53 K/s  0.00 %  7.19 % [nfsd]
 1708 be/4 root        0.00 B/s   19.36 K/s  0.00 %  5.84 % [nfsd]
Total DISK READ: 6.28 M/s | Total DISK WRITE: 488.07 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s   19.14 K/s  0.00 % 99.99 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s   45.94 K/s  0.00 % 12.44 % [nfsd]
Total DISK READ: 5.60 M/s | Total DISK WRITE: 642.64 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s   11.61 K/s  0.00 % 99.90 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s  147.11 K/s  0.00 % 11.63 % [nfsd]
 1018 be/3 root        0.00 B/s    3.87 K/s  0.00 %  3.28 % [jbd2/dm-2-8]
Total DISK READ: 6.76 M/s | Total DISK WRITE: 543.90 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s   15.48 K/s  0.00 % 95.35 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s  135.49 K/s  0.00 % 11.20 % [nfsd]
  483 be/3 root        0.00 B/s    3.87 K/s  0.00 %  2.11 % [jbd2/dm-0-8]
Total DISK READ: 5.85 M/s | Total DISK WRITE: 509.07 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s    7.74 K/s  0.00 % 99.99 % [jbd2/md2-8]
 1702 be/4 root        0.00 B/s   54.20 K/s  0.00 %  8.87 % [nfsd]
Total DISK READ: 6.52 M/s | Total DISK WRITE: 530.46 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1021 be/3 root        0.00 B/s    7.74 K/s  0.00 % 94.30 % [jbd2/md2-8]
 1704 be/4 root        0.00 B/s   85.18 K/s  0.00 %  7.22 % [nfsd]
 1702 be/4 root        0.00 B/s    7.74 K/s  0.00 %  5.92 % [nfsd]

== NFS Server ==

kernel 2.6.32-279.11.1.el6.x86_64

Here is my exports:
/srv/stockage/devel	192.168.2.0/24(rw,insecure,fsid=0)

vm.dirty_background_ratio = 10
vm.dirty_background_bytes = 0
vm.dirty_ratio = 20
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 3000

dd inside the export directory shows ~100MB/s 

== NFS Client ==

kernel 3.6.2-4.fc17.x86_64

On my client, here are my mount options:
stockage:/ on /mnt/stockage type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.30,local_lock=none,addr=192.168.2.111)

vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500

with the async export option the write rate increase from 80KBps to 4MBps! It's more usuable with this option.

(In reply to comment #19)
> I think I may have the same problem.

I see no evidence that this is the same problem.  Please file a separate bug, and if it does turn out to have the same root cause then we'll mark it as a dup later.

Ok thanks, bug #869260 has been opened

This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 17 changed to end-of-life (EOL) status on 2013-07-30. Fedora 17 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.