Bug 513006 - "allow_tar" option does not work
Summary: "allow_tar" option does not work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: viewvc
Version: 10
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Bojan Smojver
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-07-21 16:21 UTC by Patrick Monnerat
Modified: 2009-08-12 20:54 UTC (History)
2 users (show)

Fixed In Version: 1.0.9-1.fc10
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-08-12 20:54:15 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Dirty patch to fix this bug (733 bytes, patch)
2009-07-30 18:29 UTC, Patrick Monnerat
no flags Details | Diff

Description Patrick Monnerat 2009-07-21 16:21:23 UTC
Description of problem:
When option "allow_tar" is set to 1, the link to generate a tarball appears, but clicking on it results in:
-------------------------------------------
An Exception Has Occurred
Python Traceback

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/viewvc/lib/viewvc.py", line 3753, in main
    request.run_viewvc()
  File "/usr/lib/python2.5/site-packages/viewvc/lib/viewvc.py", line 406, in run_viewvc
    self.view_func(self)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/viewvc.py", line 2987, in download_tarball
    generate_tarball(fp, request, [], [])
  File "/usr/lib/python2.5/site-packages/viewvc/lib/viewvc.py", line 2969, in generate_tarball
    generate_tarball(out, request, reldir + [file.name], stack, mtime)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/viewvc.py", line 2969, in generate_tarball
    generate_tarball(out, request, reldir + [file.name], stack, mtime)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/viewvc.py", line 2969, in generate_tarball
    generate_tarball(out, request, reldir + [file.name], stack, mtime)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/viewvc.py", line 2950, in generate_tarball
    out.write(contents)
IOError: [Errno 32] Broken pipe
-------------------------------------------

In the same time, /var/log/httpd/error_log gets:
-------------------------------------------
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.5/site-packages/viewvc/lib/popen.py", line 318, in run
    self.destfile.write(s)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py", line 101, in write
    self.server.write(s)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py", line 83, in __getattr__
    return getattr(self.self(), key)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py", line 80, in self
    return self.__dict__['servers'][thread.get_ident()]
KeyError: -1214313584


gzip: stdout: Broken pipe
-------------------------------------------

Version-Release number of selected component (if applicable):
viewvc-1.0.8-1.fc10
mod_python-3.3.1-8
python-2.5.2-1.fc10
httpd-2.2.11-2.fc10


How reproducible:
Always

Steps to Reproduce:
1. Configure viewvc for an svn repository. Set "allow_tar = 1"
2. Point a browser according to the above config, into the repository
3. Click "Download GNU tarball"
  
Actual results:
See above.

Expected results:
Download of a tarball.

Comment 1 Bojan Smojver 2009-07-27 02:09:15 UTC
Hmm, not sure. I'm not really a Python guy, just packaged this.

Out of curiosity, have you tried the new 1.1.1 release by any chance (it is available in F-11)? Does that work better?

Comment 2 Patrick Monnerat 2009-07-27 16:51:25 UTC
I just recompiled the F11 1.1.1 release for F10 and it works. Now the problem is: you have to merge your old configuration into the new config file: they are not 100% compatible.
By the way: this does not really fix the F10 distro...
Thanks for the suggestion

Comment 3 Bojan Smojver 2009-07-27 21:52:10 UTC
If we release 1.1.1 as an update to F-10, would that be OK for you?

Comment 4 Patrick Monnerat 2009-07-28 08:27:58 UTC
Personnally: Yes! But I fear people not aware of this problem will get in trouble by a simple "yum update" installing a release that is not 100%compatible with their current configuration file. Whether to do it or not is under your own responsibility: As a reference, clamav people did not do it, but mediawiki people did!
The best solution would be to fix the current F10 version, but like you, I'm not a Python guy so I can't help you for that. Sorry.

Comment 5 Bojan Smojver 2009-07-29 03:04:27 UTC
Looking at the error, I'm guessing this may have something to do with permission to write to whatever turns out to be the target tarball directory (most likely directory is read only for apache user).

I'm wondering if we need to modify the code to generate the tarball in /var/spool/viewvc, like we do for enscript temp files?

Could I ask you a favour to strace httpd when the failure occurs? I just want to see two things:

1. Where does that tarball get written out to (i.e. look for open calls)?
2. Does the call to open() or write() fail with some kind of permissions issue?

You can run Apache with "httpd -X" which will then give you a single process, instead of many. Just attach yourself to it with strace -f -p `pgrep httpd` and you should see that.

Comment 6 Patrick Monnerat 2009-07-29 12:31:24 UTC
I did what you ask for and, I'm afraid, it seems there is no temporary directory and/or file: tar format is generated by python procedures and compression is achieved directly to the standard output via a pipe + fork to a gzip parallel process.
The write() fails with EPIPE, not EPERM.
I will try to investigate more from the strace output. If you want it, I can send it to you by e-mail: for confidentiality reasons (there may be auth info and/or private data in it), I will not attach it here: just ask if you want so.

Comment 7 Bojan Smojver 2009-07-29 12:55:04 UTC
OK, thanks for that info.

You are not running SELinux on this machine, right?

Comment 8 Patrick Monnerat 2009-07-29 13:00:07 UTC
No, it is disabled.

Comment 9 Bojan Smojver 2009-07-29 13:10:05 UTC
OK, thanks. Back to the drawing board.

Comment 10 Patrick Monnerat 2009-07-29 14:51:09 UTC
I found something strange in the strace output:

[pid 25568] read(18,  <unfinished ...>
 .
 . (other pids)
 .
[pid 25567] write(1, "\37\213\10\0\0\0\0\0\0\3\354\375{w\0347\2228\210\316\277\355OQ+\317\375\215\275#S\225\211"..., 16384 <unfinished ...>
[pid 25568] <... read resumed> "\37\213\10\0\0\0\0\0\0\3\354\375{w\0347\2228\210\316\277\355OQ+\317\375\215\275#S\225\211"..., 1024) = 1024
[pid 25568] close(18)
 .
 .
 .
[pid 25568] write(2, "Exception in thread Thread-1:\nTraceback (most recent call last):\n  File \"/usr/lib/python2.5/threading.py\", line 486, in __bootstrap_inner\n    self.run()\n  File \"/usr/lib/python2.5/site-packages/viewvc/lib/popen.py\", line 318, in run\n    self.destfile.write(s)\n  File \"/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py\", line 101, in write\n    self.server.write(s)\n  File \"/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py\", line 83, in __getattr__\n    return getattr(self.self(), key)\n  File \"/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py\", line 80, in self\n    return self.__dict__['servers'][thread.get_ident()]\nKeyError: -1212183664\n\n"..., 647) = 647
[pid 31390] _exit(0)                    = ?


pid 25568 executes procedure run in class _copy (/usr/lib/python2.5/site-packages/viewvc/lib/popen.py, line 312)

  def run(self):
    try:
      while 1:
        s = os.read(self.srcfd, 1024)
        if not s:
          break
        self.destfile.write(s)
    finally:
      os.close(self.srcfd)

This means there is an exception in this thread at write time. It should be read as:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.5/site-packages/viewvc/lib/popen.py", line 318, in run    self.destfile.write(s)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py", line 101, in write
    self.server.write(s)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py", line 83, in __getattr__
    return getattr(self.self(), key)
  File "/usr/lib/python2.5/site-packages/viewvc/lib/sapi.py", line 80, in self
    return self.__dict__['servers'][thread.get_ident()]
KeyError: -1212183664

Then the thread stops, causing the gzip process exit by EPIPE, in turn causing an EPIPE error in the viewvc writing thread.
We do not see this error in the normal output because it is in a parallel (probably /dev/null-redirected?) thread. However it seems to be the cause of the displayed error.

I hope it helps

Comment 11 Bojan Smojver 2009-07-29 21:40:35 UTC
Thanks for this. Unfortunately way beyond my Python, so I think I'm going to toss that upstream to see what they make of it.

Comment 12 Patrick Monnerat 2009-07-30 18:29:58 UTC
Created attachment 355717 [details]
Dirty patch to fix this bug

OK, although I'm not used to Python, I've understand what happens and managed to get it working with a dirty python-newbie patch !!!
Module sapi.py provides classes for different kind of servers (CGI, ASP and ModPython). In our case, ModPython is used and it's derived from a ThreadedServer class that registers the Server instances associated with a particular thread.
In popen.py, a _copy class creates a thread that is NOT a server by itself, and did NOT register its Server instance. This would be OK for a non-ThreadedServer derived server class (providing we could use threads in a non-ThreadedServer ;-), but bad for our case because the ThreadedServer instance to use is determined in some procedures via the thread ident as key to the above mentioned registry. No registering --> no key --> Exception
The patch causes the _copy instance thread to register to its destination file's server, only if registering methods exist...
Probably very dirty, but it works !
Please note this code is totally suppressed in version 1.1.1

Comment 13 Bojan Smojver 2009-07-30 23:57:25 UTC
That patch is infinitely better than any I could come up with (which is none), so I may package it up and put it up for testing. Thanks for going in so much depth for this!

Comment 14 Fedora Update System 2009-08-11 00:00:02 UTC
viewvc-1.0.8-2.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/viewvc-1.0.8-2.fc10

Comment 15 Fedora Update System 2009-08-11 22:31:24 UTC
viewvc-1.0.8-2.fc10 has been pushed to the Fedora 10 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update viewvc'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F10/FEDORA-2009-8455

Comment 16 Fedora Update System 2009-08-11 23:14:43 UTC
viewvc-1.0.9-1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/viewvc-1.0.9-1.fc10

Comment 17 Fedora Update System 2009-08-12 20:54:09 UTC
viewvc-1.0.9-1.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.