Bug 1569868 - Browsing samba shares using gvfs is very slow [NEEDINFO]
Summary: Browsing samba shares using gvfs is very slow
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: gvfs
Version: ---
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Ondrej Holy
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On: 1666737
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-20 07:38 UTC by amit yadav
Modified: 2020-11-04 01:35 UTC (History)
22 users (show)

Fixed In Version: gvfs-1.36.2-9.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 01:34:13 UTC
Type: Bug
Target Upstream Version:
oholy: needinfo? (asn)


Attachments (Terms of Use)
GVFS debug log with level 10 (4.87 KB, text/plain)
2018-06-27 07:06 UTC, dfrise
no flags Details
gvfsd-samba-4.6.log (2.69 MB, text/plain)
2018-11-16 13:42 UTC, Ondrej Holy
no flags Details
gvfsd-samba-4.7.log (816.85 KB, text/plain)
2018-11-16 13:43 UTC, Ondrej Holy
no flags Details
46.pcap (22.65 KB, application/vnd.tcpdump.pcap)
2019-01-04 08:46 UTC, Ilya Basin
no flags Details
48.pcap (48.01 KB, application/vnd.tcpdump.pcap)
2019-01-04 08:51 UTC, Ilya Basin
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3419431 0 None None None 2018-05-21 12:55:00 UTC
Red Hat Product Errata RHSA-2020:4451 0 None None None 2020-11-04 01:34:32 UTC

Description amit yadav 2018-04-20 07:38:58 UTC
Description of problem:
Access to samba shares is very slow after RHEL7.5 package update. The problem is reproducible in GUI mode while accessing the share using Nautilus. 
The problem is also reproducible with gvfs-mount command in text mode.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 7.5
samba-client-4.7.1-6.el7.x86_64
samba-client-libs-4.7.1-6.el7.x86_64 

How reproducible:
Always

Steps to Reproduce:
1. Apply RHEL7.5 packages update on RHEL7.4 system which is configured to access samba shares.
2. Relogin into the system and try to access samba shares using nautilus in GUI mode, or Access shares using gvfs-mount.

Actual results:
Samba share access is very slow

Expected results:
Samba share access should work normal.

Additional info:

Performance comparison between cifs vs gvfs-smb mounts (directory contains 224 entries)

1. Directory listing times after "# mount -t cifs //SERVERNAME/SHARE /mnt -o username=USER,domain=DOMAIN" 

]# cd /mnt/DIR/SUBDIR1/
]# time ls -la
...
-rwxr-xr-x. 1 root root  3446 Jan 11  2017 a.cfg
-rwxr-xr-x. 1 root root 10819 Nov 29  2013 b.cfg
-rwxr-xr-x. 1 root root  3064 Mar 23  2016 c.cfg

real	0m0.010s
user	0m0.003s
sys	0m0.004s

2. Directory listing times with the same share mounted with the "Browse Network" application (Connect to Server).

]$ cd /run/user/12184/gvfs/smb-share\:server\=SERVERNAME\,share\=SHARE/

smb-share:server=SERVERNAME,share=SHARE]$ cd DIR1/SUBDIR1/

DIR1/SUBDIR1/]$ time ls -la
....
-rwx------. 1 dfrise unil-empl-g  3446 Jan 11  2017 a.cfg
-rwx------. 1 dfrise unil-empl-g 10819 Nov 29  2013 b.cfg
-rwx------. 1 dfrise unil-empl-g  3064 Mar 23  2016 c.cfg

real	0m11.597s
user	0m0.000s
sys	0m0.015s

Comment 3 Andreas Schneider 2018-06-14 13:33:43 UTC
Could you get us a log level 10 debug log, so that we see what is going on. This looks like it is running into a connection timeout.

See GVFS_SMB_DEBUG at

https://wiki.gnome.org/Projects/gvfs/doc

Comment 4 dfrise 2018-06-27 07:06:03 UTC
Created attachment 1454937 [details]
GVFS debug log with level 10

Comment 5 Andreas Schneider 2018-10-15 13:44:25 UTC
Ondrej, could you help finding what the issue is? I have no idea how to debug gvfs to see where and why it is so slow.

Comment 6 Ondrej Holy 2018-10-16 09:42:59 UTC
Hmm, I've already seen many bug reports regarding samba backend performance, but if I recall correctly, the problem always was file transfer speed, not directory listing. But if samba downgrade helps to fix this issue, I don't think this is gvfs issue. It would be nice to try with some other libsmbclient-based tools.

Is `smbclient` also so slow (though not sure that this really uses the libsmbclient methods)? It would be good to compare with KIO also...

smbclient //SERVERNAME/SHARE -U USER -W DOMAIN -c "ls DIR1/SUBDIR1/*"

Enumeration job consist of just `opendir`, `getdents`, `stat` and `closedir` libsmbclient methods, see:
https://gitlab.gnome.org/GNOME/gvfs/blob/master/daemon/gvfsbackendsmb.c#L1734

However, `ls /run/user/12184/gvfs/smb-share\:server\=SERVERNAME\,share\=SHARE/` uses fuse mount, which may also call other jobs. Can you please try it directly over `gvfs-ls`? Is `gvfs-ls` also similarly slow?

gvfs-mount smb://SERVERNAME/SHARE
gvfs-ls smb://SERVERNAME/SHARE/DIR1/SUBDIR1

The attached gvfs debug output, unfortunately, doesn't contain samba debug info for some reason. Can you please try again e.g. using the steps from https://wiki.gnome.org/Projects/gvfs/debugging#Getting_debug_logs and use the following as the second step and ideally only `gvfs-mount` and `gvfs-ls` as the third step:

GVFS_SMB_DEBUG=10 GVFS_DEBUG=1 $(find /usr/lib* -name gvfsd 2>/dev/null) --replace 2>&1 | tee gvfsd.log

Comment 7 Ondrej Holy 2018-11-16 13:40:35 UTC
I am able to reproduce some slowdown using RHEL 7.4 VM and folder with 10000 files against server on F29 VM on the same host. I suppose that it would be much more obvious with some remote server even with smaller amount of files. It is reproducible only by updating samba packages, so changing component to samba...

$ rpm -q libsmbclient
libsmbclient-4.6.2-8.el7.x86_64
$ time gio list -la "*" smb://192.168.122.252/user/test > /dev/null
real	0m4.619s
user	0m0.901s
sys	0m0.038s
$ time ls -la /mnt/test > /dev/null
real	0m1.733s
user	0m0.064s
sys	0m0.243s

$ sudo yum update samba*
$ rpm -q libsmbclient
libsmbclient-4.7.1-6.el7.x86_64 
$ time gio list -la "*" smb://192.168.122.252/user/test > /dev/null
real	0m6.463s
user	0m0.854s
sys	0m0.036s
$ time ls -la /mnt/test > /dev/null
real	0m1.632s
user	0m0.058s
sys	0m0.230s

GVfs calls smbc_opendir and smbc_getdents once and smbc_stat for EACH file in case of "-la" options. It seems to me that this is because smbc_stat is slower with new versions for some reason... any idea why?

I suppose that CIFS uses some faster approach which is not available in libsmbclient and thus is generally faster than GVfs. It can't be reproduced over smbclient tool, because it simply doesn't provide "-la" options. 

I am going to attach corresponding gvfsd.log files with GVFS_SMB_DEBUG=10-

Comment 8 Ondrej Holy 2018-11-16 13:42:41 UTC
Created attachment 1506399 [details]
gvfsd-samba-4.6.log

Comment 9 Ondrej Holy 2018-11-16 13:43:11 UTC
Created attachment 1506400 [details]
gvfsd-samba-4.7.log

Comment 10 dfrise 2018-11-21 07:28:39 UTC
Hi,

For your information, under RHEL 7.5 the problem was fixed after upgrading all related samba packages from version 4.7.1-6.el7 to 4.7.1-9.el7_5:

Aug 23 09:16:21 Updated: samba-common-4.7.1-9.el7_5.noarch
Aug 23 09:16:21 Updated: samba-common-libs-4.7.1-9.el7_5.x86_64
Aug 23 09:16:21 Updated: libwbclient-4.7.1-9.el7_5.x86_64
Aug 23 09:16:21 Updated: samba-client-libs-4.7.1-9.el7_5.x86_64
Aug 23 09:16:22 Updated: libsmbclient-4.7.1-9.el7_5.x86_64
Aug 23 09:16:22 Updated: samba-client-4.7.1-9.el7_5.x86_64

After upgrading to RHEL 7.6 and version 4.8.3-4.el7 of samba packages, the issue did not reappeared.

Dominique

Comment 11 Ondrej Holy 2018-11-21 12:13:45 UTC
Thanks for the feedback, however, I see the same slowness as described in Comment 7 also with 4.8.3-4.el7 and 4.7.1-9.el7_5, so maybe you hit an another issue than me...

Comment 12 Andreas Schneider 2018-12-13 18:36:09 UTC
To someone who could reproduce it:

I need network traces of browsing the exact same directory with Samba 4.6 and Samba 4.8 to be able to compare traces and see what is going on.

Comment 13 Ilya Basin 2019-01-02 18:00:42 UTC
(In reply to Andreas Schneider from comment #12)
> To someone who could reproduce it:
> 
> I need network traces of browsing the exact same directory with Samba 4.6
> and Samba 4.8 to be able to compare traces and see what is going on.

I can assist, though I can test it with the Archlinux packages 4.6.7 and 4.8.5.
What kind of traces do you need? Are *.pcap files sufficient?

For now, what I see is: `ls -l` on a cifs mount is satisfied with just a single request:

    SMB2_FIND_ID_FULL_DIRECTORY_INFO Pattern: *

`gio list` with smbclient 4.6 uses SMB (v1) and for some unobvious reason does one QUERY_PATH_INFO for each found file.

`gio list` with smbclient 4.8 uses SMB2 and does three requests instead of one for each found file: 

    - Create Request File
    - GetInfo Request FILE_INFO/SMB2_FILE_ALL_INFO File
    - Close Request File

And as expected, this consumes x3 time compared to 4.6

Comment 14 Andreas Schneider 2019-01-04 08:37:52 UTC
Yes, I need pcap files.

The following link has some hints about tcpdump https://hackmd.io/s/SkHk8rXBz#

Comment 15 Ilya Basin 2019-01-04 08:46:15 UTC
Created attachment 1518319 [details]
46.pcap

Comment 16 Ilya Basin 2019-01-04 08:51:12 UTC
Created attachment 1518320 [details]
48.pcap

Attached 46.pcap and 48.pcap

Client commands:

    gio mount -a 'smb://okdistrhw/distr-ro'
    time gio list 'smb://okdistrhw/distr-ro/unmountable'

Non-default server config:

	[global]
	map to guest = Bad User

	[distr-ro]
	path = /exports/okdistr
	public = yes

Comment 17 Jeremy Allison 2019-01-07 20:03:22 UTC
Ah, here is the problem.

gio is doing opendir/readdir/close to get the file
name list.

At this point we're actually already fetching all
required info to satisfy a stat via SMB2_FIND_ID_BOTH_DIRECTORY_INFO.

Then gio does a stat() on every name it got from
the readdir call.

That is what is translating into QUERY_PATH_INFO in SMB1
and open/getinfo/close in SMB2.

What gio should be doing is the readdirplus() call,
which associates the stat info we already got in
the directory traversal.

Can someone point me at the gio code so I can see
exactly what libsmbclient calls it's using here ?

Jeremy.

Comment 18 Jeremy Allison 2019-01-07 20:44:18 UTC
Found the source code for the SMB backend to gvfs. It's doing:

  while (TRUE)
    {
      res = smbc_getdents (op_backend->smb_context, dir, (struct smbc_dirent *)dirents, sizeof (dirents));
      if (res <= 0)
        break;
      
      dirp = (struct smbc_dirent *)dirents;
      while (res > 0)
        {
          unsigned int dirlen;

          /* TODO: Only do stat if required for flags */
          
          if ((dirp->smbc_type == SMBC_DIR ||
               dirp->smbc_type == SMBC_FILE ||
               dirp->smbc_type == SMBC_LINK) &&
              strcmp (dirp->name, ".") != 0 &&
              strcmp (dirp->name, "..") != 0)
            {
              int stat_res;
              g_string_truncate (uri, uri_start_len);
              g_string_append_uri_escaped (uri, dirp->name, SUB_DELIM_CHARS ":@/", FALSE);

              if (matcher == NULL ||
                  g_file_attribute_matcher_matches_only (matcher, G_FILE_ATTRIBUTE_STANDARD_NAME))
                {
                  info = g_file_info_new ();
                  g_file_info_set_name (info, dirp->name);
                  g_vfs_job_enumerate_add_info (job, info);
                  g_object_unref (info);
                }
              else
                {
                  stat_res = smbc_stat (op_backend->smb_context,
                                                            uri->str, &st);

which is as I described. It should be calling smbc_readdirplus() instead here, which would directly avoid the extra smbc_stat() call.

Comment 19 Andreas Schneider 2019-01-08 05:43:43 UTC
Thanks for your help Jeremy! Moving the bug to Ondrej.

Comment 20 Andreas Schneider 2019-01-08 06:33:49 UTC
Ondrej, the call will be available with Samba in RHEL 7.7.

Comment 21 Ondrej Holy 2019-01-08 07:21:39 UTC
I've recently found "smbc_readdirplus()" and tried to use it there, however, it seems that "struct libsmb_file_info" doesn't contain info about file type which is crucial. There is only "attrs", which can be used to distinguish a directory from a file, but not e.g. from a symbolic link...  or am I mistaken?

See:
https://gitlab.gnome.org/GNOME/gvfs/issues/306
https://gitlab.gnome.org/GNOME/gvfs/merge_requests/21

Comment 23 Ondrej Holy 2019-01-14 09:19:38 UTC
Andreas, thanks. So let's move the bug back to samba until smbc_readdirplus2 will be available.

Comment 24 Andreas Schneider 2019-01-14 15:20:48 UTC
I think this is a gvfs bug, you should open a new one for smbc_readdirplus2():

Add a new smbc_readdirplus2() function to smbclient for faster directory listing

Comment 26 Ondrej Holy 2019-08-09 07:43:00 UTC
Given the fact that the dependent samba bug hasn't been yet resolved and RHEL 7 is already in the maintenance phase, I am going to close this bug. Please make a clone for RHEL 8 if it is desirable.

Comment 27 Jeremy Allison 2019-08-09 16:25:06 UTC
Yes, please clone for RHEL 8, we are planning to address this in Samba, it's just that we're resource constrained right now.

Comment 35 Tomas Pelka 2020-08-03 12:07:08 UTC
I'm not able to reproduce with recent versions of gvfs and samba. But to be 100% sure can you please Amit ask customer to double check with 8.3.0 Beta?

Comment 36 Tomas Pelka 2020-09-03 19:02:46 UTC
(In reply to Tomas Pelka from comment #35)
> I'm not able to reproduce with recent versions of gvfs and samba. But to be
> 100% sure can you please Amit ask customer to double check with 8.3.0 Beta?

ping

Comment 42 errata-xmlrpc 2020-11-04 01:34:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: GNOME security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4451


Note You need to log in before you can comment on or make changes to this bug.