Bug 585758

Summary: unable to create files on WebDAV fs (but mkdir worked fine)
Product: [Fedora] Fedora Reporter: Tim Taiwanese Liim <tim.liim>
Component: davfs2Assignee: Will Woods <wwoods>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 12CC: wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-04-30 04:27:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tim Taiwanese Liim 2010-04-25 23:29:37 UTC
Description of problem:
    In F12 as root I mounted a WebDAV share with mount.davfs, was able to
    read files, mkdir, chdir; but was not able to create file, even as
    root, with this error:
        touch: cannot touch `xxx: Input/output error

Version-Release number of selected component (if applicable):
    davfs2-1.3.3-3.fc12.x86_64

How reproducible:
    always

Steps to Reproduce:
    1.login as root
    2.m=/media/t
      mkdir -p $m
      mount -t davfs http://tacec.org:2077/ $m
      <enter login/passwd>
    3.mkdir $m/x1
    4.touch $m/x2
  
Actual results:
    step 3 completed without error, and ls -l showed x1.
    step 4 has this error msg
        touch: cannot touch `/media/tacec/x2: Input/output error

Expected results:
    step 4 should also succeed.

Additional info:
    - I can provide the TCP trace if needed.
    - the https counterpart also has the same error, but I figured
      it's easier to debug with http.

Comment 1 Tim Taiwanese Liim 2010-04-25 23:43:53 UTC
Tested with 
  davfs2-1.4.5-2.fc13.i686 
just now.
  - mkdir still works
  - touch $m/x2 fails with this msg after >10 sec timeout:
        touch: cannot touch `/media/t/x2': Resource temporarily unavailable

Comment 2 Will Woods 2010-04-26 17:53:58 UTC
It's hard to say why this might be failing without knowing anything about the server. But I checked the url you mention in comment #1 and it seems to be a cPanel WebDisk. 

This server uses a WebDAV implementation that is known to have serious problems, including a completely broken locking implementation. See the davfs2 support tracker for some details: http://savannah.nongnu.org/support/?106922

As noted in that support ticket, the folks at cPanel are aware of the problem. You can see a discussion of the problem on the cPanel forums: http://forums.cpanel.net/f5/webdisk-mounted-ubuntu-8-10-a-110265.html

There is a workaround mentioned in the support ticket and forum thread: "add a line with "use_locks 0" to your davfs2.conf file." Could you try that and see if it fixes the problem for you?

Comment 3 Tim Taiwanese Liim 2010-04-27 01:46:30 UTC
Will,
You are right on for all counts.  Appreciate your kind help!
- yes, my ISP is using cPanel WebDisk.
- yes, agree that cPanel should not say "LOCK is supported" 
  while it does not.
- yes, adding "use_locks 0" fixed my problem.

But... then I found new problems.
1. writing to a file make it zero length:
   so I can create new files now, eg. 
        echo test > test
   and "ls -l" shows the file with 5 bytes; umount webdav and
   then re-mount, and now the file size of 'test' is 0 byte.

2. very slow, in the range of 1kB/sec or less:
   create a file of 5 bytes on the webdav share; the umount takes > 1
   minute to complete.  FTP 100kB to same host take <0.5 sec.
   Wireshark shows that the TCP stream stalls often: davfs2 would
   write a few hundred bytes, receives TCP ACK from server, but then
   davfs2 would do nothing for 30 sec, then close the TCP connection.

Any idea?

Comment 4 Tim Taiwanese Liim 2010-04-29 14:43:41 UTC
Re: davfs2 appears to stall on write

I used nautilus (dav://tacec.org:2077/) to write a file, and it seems
to be fine.  The related section from Wireshark has (the content of
the file "test" is "test")
    PUT /test HTTP/1.1
    Host: tacec.org:2077
    User-Agent: gvfs/1.4.3
    Authorization: Basic <credential censored>
    Content-Length: 5

    test
to which the server responded "HTTP/1.1 201 CREATED" immediately.


Then I used davfs2 for a similar write (the content of the file "test"
is "test"):
    PUT /2010/1997/test HTTP/1.1
    User-Agent: davfs2/1.3.3 neon/0.29.3
    Connection: TE
    TE: trailers
    Host: tacec.org:2077
    Content-Length: 5
    Expect: 100-continue
    Authorization: Basic <credential censored>

Note that davfs2 did *not* send the content of the file, the length of
which is 5.  After this, server did not send response (probably
waiting for the content).  30 sec later, davfs2 timed out and closed
TCP connection.

Is this an issue in davfs2?  Shouldn't davfs2 send the content of
file as part of the PUT request?  I'll compare the trace when
connecting to apache2, because when I tested davfs2-apache2 a few
days ago it worked fine, no stall in create a new file.

Comment 5 Will Woods 2010-04-29 18:32:46 UTC
Sounds like a davfs2 bug to me. You should probably file it with upstream, since I'm just a humble packager, not actually a davfs2 developer. 

http://savannah.nongnu.org/projects/davfs2

It's probably a good idea to link back to this bug report, too.

Comment 6 Tim Taiwanese Liim 2010-04-30 04:25:42 UTC
Will,
You are more than just a packager; you are responsive and helpful,
pointing me into the right direction.  Thanks a lot for your kind help
and fast response!

After some more debugging, I think this is just an unfortunate
inter-operability issue between davfs2-cPanel, which can be fixed by
adding these in davfs2.conf:
    #1 use_locks     0
    #2 use_expect100 0  # see [1]

Reason for #1: See Comment #2 by Will.
Reason for #2:

I checked the trace between davfs2 and apache2, which worked fine, as
illustrated in [2], because apache2 responds to "Expect: 100-continue"
properly.  But cPanel does not honor "Expect: 100-continue", as
illustrated in Comment #4, so davfs2 times out.  The fix is #2,
telling davfs2 not to use expect100.

[1] stated "not all servers understand this;" apparently cPanel is one
of them.  Is it cPanel's fault not to honor expect100?  I checked the
RFC for WebDAV [3][4] and could not find reference to "expect" or
"continue", so I guess cPanel can argue that the standard does not
require expect100.  

After adding #1, #2, davfs2-cPanel works reasonably fast for me.  So I
will close this ticket.

----------------------------------------------------------------------
[1] man davfs2.conf
    use_expect100
      To avoid uploading big files that will be refused by the server,
      mount.davfs uses the header expect: 100-continue to get the o.k.
      from the server before uploading. Not all servers understand
      this.  0 = no, 1 = yes.
      Default: 1


[2] wireshark trace between davfs2 and apache2
    davfs2
        PUT /webdav/test1 HTTP/1.1
        User-Agent: davfs2/1.3.3 neon/0.29.3
        Connection: TE
        TE: trailers
        Host: qyam.yam
        Content-Length: 8
        Expect: 100-continue    <== davfs2 expectes 100-continue
        Authorization: Basic <credential censored>
    apache2
        HTTP/1.1 100 Continue   <== apache2 says 100 Continue
    davfs2
        test123
    apache2
        HTTP/1.1 201 Created
        ...

[3] RFC 4918: HTTP Extensions for Web Distributed Authoring and
    Versioning (WebDAV)

[4] RFC 2518: HTTP Extensions for Distributed Authoring -- WEBDAV

Comment 7 Tim Taiwanese Liim 2010-04-30 04:26:33 UTC
One more issue: in davfs2, when I tries to change the file time stamp
with touch, eg.
    touch -d 1/1/2010 test
where test is an existing file, the timestamp is not persistent across
umount/mount.  Right after touch, the ts is "1/1/2010", as desired.
Then I umount and mount again, now the ts become 4/30/2010.  This is
the same for both apache2 and cPanel, so I guess this is a constraint
of WebDAV, and nothing I can do to fix it.  

But if any knows how to do this, please let me know.  Thanks.