Bug 406871

+++ This bug was initially created as a clone of Bug #344851 +++

Description of problem:
The juju firewire stack lacks proper full support for ohci 1.0-based firewire
controllers. Most glaringly, the lack of isochronous I/O support means dv
cameras are non-functional with 1.0-based controllers.

When we follow the call chain from dvgrab, it traces down to the call
ohci_allocate_iso_context() in drivers/firewire/fw-ohci.c, where we have:

        /* FIXME: We need a fallback for pre 1.1 OHCI. */
        if (callback == handle_ir_dualbuffer_packet &&
            ohci->version < OHCI_VERSION_1_1)
                return ERR_PTR(-EINVAL);

As Stefan mentioned in bug 328011, and as suggested by the above code, its a
known issue, listed at both http://wiki.linux1394.org/JujuMigration and
http://wiki.linux1394.org/ToDo as a top priority item.

1.0 controllers appear to be quite a bit more prevalent than first thought (in
my own systems, they outnumber 1.1 controllers 5 to 1), so we really need to get
proper support for 'em sooner than later. Attached is a patch from Stefan to at
least log the reason for the failure in the interim.

-- Additional comment from jwilson on 2007-10-21 22:35 EST --
Created an attachment (id=233701)
Stefan's patch to have firewire stack log something about not supporting 1.0
controllers


-- Additional comment from stefan-r-rhbz.de on 2007-10-22 03:38
EST --
> 1.0 controllers appear to be quite a bit more prevalent than first
> thought (in my own systems, they outnumber 1.1 controllers 5 to 1)

My own sample:
   1.0 chip types : 1.1 chip types  =  3 : 2
        1.0 cards : 1.1 cards       =  5 : 3
All acquisitions during the past 12 months (2 new, 1 used) happened to be 1.0. 
It shows that I don't do anything isochronous.

-- Additional comment from nyh.ac.il on 2007-10-28 11:35 EST --
I have just installed Fedora 7 on a new computer, and dvgrab from firewire
doesn't work, and all signs point to this bug (apparently, despite being a new
card, it is "OHCI 1.0" which isn't supported by the "Juju" patch that Fedora
used in the kernel).

To me, this looks like a major Fedora bug (more serious than the "medium
severity" assigned to it). It prevents people from using their hardware, and
worst of all, and is quite difficult to fix by manual compilation (requires
recompiling the kernel, and applications like dvgrab or libraries lib libraw1394
which were also modified by Fedora to fit the Juju patch). I was most saddened
by the rumors I've been reading (but correct me if I'm wrong) that this
situation won't be fixed in Fedora 8.

I wonder, if half the people (or whatever - see statistics in above comments)
with firewire cannot use it AT ALL with this juju patch, how on earth can it be
considered an improvement over the old ieee1394 stack? What is preventing fedora
from *temporarily* but *immediately* reverting back to kernels with the old
stack, and programs like libraw1394 unpatched, and only returning to this juju
version when it actually works for more than half the people?

Since people have bought their firewire card for a good reason (usually, to
capture video), often the card will be more important to them than their choice
of distro, and many will switch to some other distro. That would be a real
shame, and simply unnecessary if Fedora only reverted to the working version of
the kernel.


-- Additional comment from stefan-r-rhbz.de on 2007-10-28 12:51
EST --
> is quite difficult to fix by manual compilation (requires recompiling
> the kernel, and applications like dvgrab or libraries lib libraw1394
> which were also modified by Fedora to fit the Juju patch).

AFAIU dvgrab is unchanged.  Changed were kernel modules, the low-level libraries
libraw1394 and libdc1394, installation tools, and kernel-releated stuff like
initrd, perhaps hal (I'm not sure of the latter).

I can't speak for the Fedora project (I don't use Fedora and don't develop for
it, I'm just lurking here as co-maintainer of kernel.org's FireWire drivers),
but better than reverting to old drivers would be to get someone to implement
the missing functionality.  Alas I can't lend a hand at the moment due to lack
of spare time.

> I wonder, if half the people (or whatever - see statistics in above
> comments) with firewire cannot use it AT ALL with this juju patch,
> how on earth can it be considered an improvement over the old
> ieee1394 stack?

The number of OHCI 1.0 installations and the fact that even newly sold cards and
mainboards may contain OHCI 1.0 chips has alas been underestimated by those
involved, including me.  Those who use FireWire only for storage devices are not
affected by this, BTW.

-- Additional comment from stefan-r-rhbz.de on 2007-10-28 13:44
EST --
From the don't-do-this-at-home department:

Krzysztof Halasa found that VIA VT6307 can be programmed to support either OHCI
1.0 or OHCI 1.1:  http://marc.info/?l=linux1394-user&m=119292265030931

My impression though is that most VT6307 are already sold programmed in OHCI 1.1
mode.  Somebody has yet to try whether this also works with VIA VT6306; VIA's
documentation is ambiguous in that regard.  It seems to me that all VT6306 are
sold as OHCI 1.0 controllers.

-- Additional comment from nyh.ac.il on 2007-10-28 17:25 EST --
Hi Stefan, according to lspci, I have a "FireWire (IEEE 1394): VIA Technologies,
Inc. IEEE 1394 Host Controller (rev 46)" - I don't know whether or not it's a
"VT6306", and even if it were, I'm not likely to try something its own author
says "might blow up your machine" and "use at your own risk". I don't even know
for sure that my card is "OHCI 1.0", or how to check it.

Anyway, I understand why ultimately, the "firewire" ("juju") drivers are better
than the ieee1394, and why they shouldn't be completely abandoned. But I
completely fail to understand why, if the new drivers don't actually work for
the majority of the people, Fedora doesn't issue an update to the kernel and
libraw1394 (or whatever needs to be changed) returning the old drivers - which
the rest of the Linux world besides Fedora - appears to be using. When the
missing "OHCI 1.0" support is returned - hopefully more sooner than later - a
new update to the kernel and libraw1394 can return back the improved Juju kernel.

While this unfortunate situation (where half the users of firewire digital-video
grabbers can't use them on Fedora) has lasted for half a year now, I believe
that it still can be rectified (by returning the old drivers) in a very short
time, and I hope that the powers that be decide to do so.

-- Additional comment from stefan-r-rhbz.de on 2007-10-28 18:10
EST --
> I don't even know for sure that my card is "OHCI 1.0", or how to check it.

"dmesg | grep firewire_ohci" will display the OHCI revision, unless there were
already too many other kernel messages after firewire-ohci's initialization
message.  In that case, you can repeat the initialization with "modprobe -r
firewire-ohci && modprobe firewire-ohci && dmesg | grep firewire_ohci".  (Grep
needs an underscore _ in the driver name.)

The VIA rev 46 controller is most certainly a VT6306 which I believe does not
implement OHCI 1.1 and I suspect cannot be reprogrammed to do so.

The problem with the new drivers is:

In OHCI 1.1, there exist three modes for isochronous reception, all operating on
series of memory buffers:  Packet-per-buffer, which puts not more than one
packet into each buffer; buffer-fill, which concatenates packet in the buffers;
 and dual-buffer, which is like buffer-fill but with two strings of buffers. 
With dual-buffer, the controller takes each packet apart and puts all the first
parts in the first string of buffers and the second parts into the second
string.  So, regarding functionality, dual-buffer is a superset of buffer-fill.
 It is useful to separate protocol headers from payload data.  In OHCI 1.0, only
packet-per-buffer and buffer-fill were specified.  (This was only discovered
after the fact because the OHCI 1.1 spec doesn't say what's new, and the OHCI
1.0 spec is nowhere available for download.)

The firewire-ohci driver is specialized on dual-buffer mode.  What needs to be
done is to use buffer-fill mode instead, at least if an OHCI 1.0 chip is
encountered.  This requires someone with (a) knowledge of DMA programming in the
Linux kernel, (b) some insight into isochronous FireWire applications and
libraries, (c) time to implement it and hardware to test it.  At the moment I
lack at least (b) and (c).

-- Additional comment from stefan-r-rhbz.de on 2007-10-28 18:33
EST --
> So, regarding functionality, dual-buffer is a superset of buffer-fill.

This could mean that with a bit of luck...

> This requires someone with (a) knowledge of DMA programming in the
> Linux kernel, (b) some insight into isochronous FireWire applications
> and libraries, (c) time to implement it and hardware to test it.

...(b) might not be a strict requirement, and (a) could be done the
monkey-see-monkey-do way based on the existing dual-buffer code and the OHCI 1.1
spec (see download links at http://wiki.linux1394.org/Links/Specifications). 
Unless there is something more to it of which I am not yet aware of.

-- Additional comment from jwilson on 2007-10-28 23:10 EST --
I believe (b) should be mostly unnecessary, and yeah, I think (a) can mostly be
figured out by inspecting 
the dual-buffer code. I've got (c), and have starting trying to whip something
up, simultaneous with 
digesting the specs -- initially, looking at doing a buffer-fill iso receive for
ohci 1.0. I've talked to Kristain 
a little bit about all this as well, and will be picking his brain as I try to
get this working. Sounds like the 
most fun might be dissecting the packet headers from the data in software
(essentially doing what dual-
buffer does for us in hardware).

-- Additional comment from stefan-r-rhbz.de on 2007-10-29 03:02
EST --
> the most fun might be dissecting the packet headers from the data in
> software (essentially doing what dual-buffer does for us in hardware)

Does the stack actually have to do so if serving software written for raw1394?

-- Additional comment from jwilson on 2007-10-29 12:00 EST --
Good question. At a glance, I'm not seeing the dissection being done in-kernel
in the old stack, but I've not looked thoroughly. I'll peruse both the old stack
and the libraw1394 support for the old stack to see if I can't figure that
out... Certainly sounds reasonable to just hand off the raw captured data to
libraw1394 though.

-- Additional comment from stefan-r-rhbz.de on 2007-10-29 12:46
EST --
Does libraw1394's Juju backend let the kernel split off anything at all?  I
guess the answer lies in the values provided with struct
fw_cdev_create_iso_context.header_size (and struct fw_cdev_iso_packet.control's
HEADER_LENGTH part).

-- Additional comment from jwilson on 2007-10-29 17:21 EST --
Kristian suggests that libraw1394 shouldn't have to know what IR method we're
using, and we ought to present everything to libraw1394 the same way, regardless
of whether we had to go to a fallback IR method for ohci 1.0.

Thus far, I've added analogs to ohci_queue_iso_receive_dualbuffer and
handle_ir_dualbuffer_packet, unimaginatively named
ohci_queue_iso_receive_bufferfill and handle_ir_bufferfill_packet, wired them
up, and at least got the controller receiving and acknowledging packets.

The primary thing that is left (and still a bit over my head) is splitting up
the headers and data, and sending them out to userspace in an intelligible form.

-- Additional comment from jwilson on 2007-11-03 23:31 EST --
Just a quick status update... We made some pretty good progress this past week
on ohci 1.0 support, and 
for the most part, all that is left is properly sorting out the headers and
payloads into the right buffers 
where userspace expects them (so that we're presenting exactly the same thing to
userspace, regardless 
of the controller being ohci 1.1 or 1.0). Don't want to promise anything, but
I'm hoping to be grabbing 
video off my cable box w/my ohci 1.0-equipped Mac Mini by next weekend... :)

-- Additional comment from jprenaud on 2007-11-12 05:15 EST --
Any update on this? Also, would that be made available for fedora 7 too?

-- Additional comment from jwilson on 2007-11-12 09:58 EST --
Still working on it, and inching closer to completion. I believe I've got the
packet parsing working correctly now (including all the fun cases like when a
packet overflows to another buffer and/or page), but we still need to wire up
the bits to actually feed the split headers and payloads out to userspace.
Should be ready Real Soon Now(tm)... :)

And yes, once working, this will be added to a Fedora 7 update kernel as well.

-- Additional comment from choke on 2007-11-12 11:01 EST --
Awesome to head that, Jarod. Looking forward to it.

-- Additional comment from jwilson on 2007-11-13 16:12 EST --
Still some rough edges to finish up, but with (a lot of) help from Kristian this
afternoon, we just got 'dvgrab -d 5' to capture a 5 second clip using an OHCI
1.0 controller. :)

-- Additional comment from jprenaud on 2007-11-13 16:45 EST --
Great news! Thanks for your hard work. I am looking forward to kino working on F7.

-- Additional comment from jwilson on 2007-11-14 00:43 EST --
Created an attachment (id=257711)
FireWire OHCI 1.0 Iso Receive support patch

I've got a scratch kernel build with an initial cut of OHCI 1.0 Isochronous
Receive support working its 
way through the build system right now:

http://koji.fedoraproject.org/koji/taskinfo?taskID=240966

Would appreciate testing feedback from folks (i.e., does 'dvgrab -d 5' give you
a 5-second clip that 
looks and sounds sane, and results on how kino behaves -- still need to try
that myself). I'm already 
anticipating that people are going to see dvgrab spitting errors about buffer
underruns, as I've seen 
that myself on most of my test captures thus far.

Unfortunately, it seems this is largely due to this not being a particularly
efficient implementation, 
which is a result of trying to make it feed the exact same results back to
userspace as OHCI 1.1 
controllers do (so that userspace isn't required to know which one we're
doing). We actually ended up 
scrapping buffer-fill IR mode for now, instead using packet-per-buffer for OHCI
1.0, as it turned out to 
be next to impossible to make bufferfill feed the same thing as the OHCI 1.1
method (dual-buffer) out 
to userspace. I'm still hoping we can streamline the packet-per-buffer code a
bit more to help improve 
the performance...

After a bit of further poking, testing and hopefully, improving, I'll rediff
against upstream, throw an SOB or two on it and ship an official patch toward
linux1394-devel.

-- Additional comment from zimny_browar on 2007-11-15 05:43 EST --
when you finish this fix? I still in Fedora 8 have this:

dvgrab -autosplit -format raw -size 0 -noavc -timestamp foo-
Found AV/C device with GUID 0x08004601044380f6
ioctl call failed, retval = -1
ieee1394io.cc:460: In function "virtual bool iec61883Reader::StartReceive()": 
"iec61883_dv_fb_start( m_iec61883.dv, channel )" evaluated to -1
ieee1394io.cc:460: errno: 38 (Function not implemented)
""     0.00 MB 0 frames
Capture Stopped


-- Additional comment from jwilson on 2007-11-15 17:08 EST --
(In reply to comment #21)
> when you finish this fix?

There was a scratch build to test out linked above in comment #20. There should
be an updates-testing kernel relatively soon carrying this patch, though the
resulting video captures are not perfect and I'm still trying to figure out why.

-- Additional comment from wwoods on 2007-11-16 13:26 EST --
Woo! I can confirm that 'sudo dvgrab -d5 -f raw' works on an OHCI 1.0
controller. There are occasional buffer underrun messages - but it doesn't
happen every time. And I guess you need mpeg2 support to play back the resulting
video, so it's slightly tricky to test in a pure-Fedora environment. 

There are some artifacts in the video. Sometimes it fails to find the camera.
And occasionally I get this message in dmesg:
firewire_ohci: context_stop: still active (0x40000411)

But other than that, it works exactly how I'd expect it to. I've never had this
working before now, so I'm not sure if those things are normal or not. Can
someone who uses this stuff in more heavy/interesting ways give that test kernel
a shot? 

-- Additional comment from jwilson on 2007-11-20 12:20 EST --
Will's experience in comment #23 is about on par with what I see myself (though
I also occasionally hit bug 370931 as well).

The first odd thing I've noticed about the artifacts is that when they show up
in a given frame, its usually in the same places within the frame as it is for
other frames in a given capture. (i.e., the artifacts always show up at the same
x/y offsets in the video when they do show up).

The other odd thing about the artifacts is that they're often either obvious
residual data from the prior frame, or a lighter or darker version of what
should be in the current frame.

I've triple-checked the code, and thus far, have come up empty for an
explanation. Its definitely not arch-specific or a cpu power thing either, since
I get the same results on an x86_64 quad 3GHz core xeon and an i686 1GHz Via C3
EPIA...

-- Additional comment from updates on 2007-11-26 13:50 EST --
kernel-2.6.23.8-34.fc7 has been pushed to the Fedora 7 testing repository.  If
problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update kernel'

-- Additional comment from jwilson on 2007-11-26 14:00 EST --
Nb: the updates-testing kernel comes with the version described in earlier
comments. Its not perfect, but better than nothing, and we're working on fixing
its deficiencies.

-- Additional comment from ahziem1 on 2007-11-27 10:57 EST --
I am trying 2.6.23.8-34.fc7.  When running as non-root, dvgrab and Kino find no
camera.  Kino complains about libraw1394 not loaded or can't read/write
/dev/raw1394.  Actually, that does not exist.  Log files for non-root:

Nov 27 07:22:51 z kernel: firewire_ohci: Added fw-ohci device 0000:00:0b.0, OHCI
version 1.0
Nov 27 07:22:51 z kernel: firewire_core: created new fw device fw0 (0 config rom
retries, S400)
Nov 27 08:39:55 z kernel: firewire_ohci: Added fw-ohci device 0000:00:0b.0, OHCI
version 1.0
Nov 27 08:39:55 z kernel: firewire_core: created new fw device fw0 (0 config rom
retries, S400)
Nov 27 08:40:54 z kernel: firewire_core: phy config: card 0, new root=ffc1,
gap_count=5
Nov 27 08:40:54 z kernel: firewire_core: phy config: card 0, new root=ffc1,
gap_count=5
Nov 27 08:40:55 z kernel: firewire_core: created new fw device fw1 (0 config rom
retries, S100)

When running "dvgrab sony-" as root, it operates the camera and creates a file,
but the captured data is corrupt.  The file is not recognized by Kino, vlc,
gmplayer, or file.  Here are the logs for root:

Nov 27 08:51:17 z kernel: firewire_ohci: context_stop: still active (0x40000411)
Nov 27 08:51:17 z kernel: dvgrab[3937]: segfault at b66fc004 eip 00ba83ec esp
bf9b1148 error 4

# lspci
00:0b.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller
(rev 80)


-- Additional comment from ahziem1 on 2007-11-27 11:00 EST --
Sorry.  After waiting for the file to finish, I can play it.  Video looks OK,
but I hear a clicking in the audio track.  I'll test more later.

-- Additional comment from stefan-r-rhbz.de on 2007-11-27 11:47
EST --
Re comment #27:

The inability to use kino and dvgrab as unprivileged user is very certainly a
matter of device file permissions:
$ ls -l /dev/fw*
These device files are used by the juju-enabled libraw1394 instead of
/dev/raw1394.  Any error messages referring to /dev/raw1394 are just remnants of
the old IEEE 1394 kernel interface.

Ideally, a facility like udev or hal would adjust the permissions of /dev/fw*
files which belong to nodes which have AV/C units or IIDC units so that users
who are for example members of a video group or something like that could read
and write to these device files.  Of course this would need explicit support by
udev or hal.

The permissions problem has also been mentioned in
https://bugzilla.redhat.com/process_bug.cgi#c23
It can be seen as successor of bug 191670.

As for the segfault:  Another dvgrab segfault is mentioned in bug 370931.  When
I tested dvgrab with an OHCI 1.1 controller on a vanilla x86-32 kernel, it
always segfaulted on exit but wrote a valid video file before that.   When I
tested dvgrab with an OHCI 1.0 controller on a vanilla x86-64 kernel with
Jarod's patch for OHCI 1.0 iso reception, dvgrab did not segfault.

Re comment #28:

When captured by dvgrab via OHCI 1.1, video and audio are fine.  When captured
by dvgrab via OHCI OHCI 1.0, video is defective like described by Jarod in
comment #24.  Audio appeared to be fine.  I tested with a JVC miniDV camcorder.

I have less success with Kino.  Except for a single lucky occasion, it always
segfaults in raw1394_iso_recv_start as soon as I attempt to enter the Capture
tab, regardless whether a camera is connected or not.

-- Additional comment from jwilson on 2007-11-30 13:19 EST --
(In reply to comment #29)
> The inability to use kino and dvgrab as unprivileged user is very certainly a
> matter of device file permissions:
> $ ls -l /dev/fw*
> These device files are used by the juju-enabled libraw1394 instead of
> /dev/raw1394.  Any error messages referring to /dev/raw1394 are just remnants of
> the old IEEE 1394 kernel interface.
> 
> Ideally, a facility like udev or hal would adjust the permissions of /dev/fw*
> files which belong to nodes which have AV/C units or IIDC units so that users
> who are for example members of a video group or something like that could read
> and write to these device files.  Of course this would need explicit support by
> udev or hal.

hal is supposed to be taking care of access permissions after a user logs into
the desktop, and does do so on a freshly installed F8 system for me. I've seen
other situations where it doesn't though -- my
originally-fc6-upgraded-to-f7-then-to-f8 Mac Mini doesn't get the access
permissions right either, though that box is running ratpoison for its window
manager (maybe no hal stuff loading at login then?).


> Re comment #28:
> 
> When captured by dvgrab via OHCI 1.1, video and audio are fine.  When captured
> by dvgrab via OHCI OHCI 1.0, video is defective like described by Jarod in
> comment #24.  Audio appeared to be fine.  I tested with a JVC miniDV camcorder.

New patch coming shortly, which on every card I've tried outside of Via VT6307
ones, produces flawless captures using an improved ohci 1.0 IR method. I got no
less than five different firewire chipsets that all captured perfectly with this
forthcoming version, but the VT6307 simply doesn't want to work for reasons I've
yet to figure out.

> I have less success with Kino.  Except for a single lucky occasion, it always
> segfaults in raw1394_iso_recv_start as soon as I attempt to enter the Capture
> tab, regardless whether a camera is connected or not.

A lot of the kino code is identical to dvgrab code, so I suspect the kino
segfault is rather similar in nature to the dvgrab one, and could be fixed in a
similar fashion.

-- Additional comment from jwilson on 2007-11-30 14:13 EST --
Created an attachment (id=274231)
Updated FireWire OHCI 1.0 Iso Receive patch for 2.6.23.x


-- Additional comment from jwilson on 2007-11-30 14:14 EST --
Created an attachment (id=274241)
Upstream version of FireWire OHCI 1.0 Iso Receive patch