Bug 528909 - DevKit 95-devkit-disks.rules should avoid scan all DM devices
Summary: DevKit 95-devkit-disks.rules should avoid scan all DM devices
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: DeviceKit-disks
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: David Zeuthen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: F12Blocker, F12FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2009-10-14 10:17 UTC by Milan Broz
Modified: 2013-03-01 04:07 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-11-03 20:44:36 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Milan Broz 2009-10-14 10:17:09 UTC
Description of problem:

The udev rule 95-devkit-disks.rules (owned by DevKit-disks) scan all device-mapper devices (including internal lvm and cryptsetup devices) by running devkit-disks-dm-export and followed by blkid.

This can cause serious problems:

1) it should not react to "add" event but only to "change" event.
(internally DM device is not yet properly configured - "add" event is generated
by block layer automatically before the DM loads proper mapping table)

2) it must never scan and open internal devices, at least
- temporary-cryptsetup
- internal lvm devices like "pvmove"

Here is example what happens:
- running cryptsetup (here intentionaly with wrong passphrase to create & destroy many temporary keyslot devices)

while :; do echo "xxc" | cryptsetup luksOpen /dev/sdg x ; done

device-mapper: remove ioctl failed: Device or resource busy
No key available with this passphrase.
No key available with this passphrase.
No key available with this passphrase.
Command failed: No key available with this passphrase.
device-mapper: remove ioctl failed: Device or resource busy
No key available with this passphrase.
device-mapper: remove ioctl failed: Device or resource busy
No key available with this passphrase.
...

(when running with --debug you can see that cryptsetup force removes device)
# dm remove temporary-cryptsetup-8867  OF   [16384]
device-mapper: remove ioctl failed: Device or resource busy
# WARNING: other process locked internal device temporary-cryptsetup-8867, retrying remove.

and it leads to very annoying messages in log (cryptsetup removes this keyslot device by remmaping it to error target)

device-mapper: ioctl: unable to remove open device temporary-cryptsetup-8867
__ratelimit: 189 callbacks suppressed
Buffer I/O error on device dm-0, logical block 0
Buffer I/O error on device dm-0, logical block 1
Buffer I/O error on device dm-0, logical block 2
Buffer I/O error on device dm-0, logical block 3
Buffer I/O error on device dm-0, logical block 4
Buffer I/O error on device dm-0, logical block 5
Buffer I/O error on device dm-0, logical block 6
Buffer I/O error on device dm-0, logical block 7
Buffer I/O error on device dm-0, logical block 8
Buffer I/O error on device dm-0, logical block 9

(The worse situation can happen for pvmove see bug 520913)

The same situation was just fixed in anaconda internal rules, see bug 526699)

I know that these rules will be obsoleted soon but unfortunately they are
now still present in F12 and I think this should be blocker for release.

Version-Release number of selected component (if applicable):

# rpm -qf 95-devkit-disks.rules
DeviceKit-disks-007-4.fc12.x86_64

Comment 1 Milan Broz 2009-10-14 10:20:46 UTC
Adding to F12 blocker.

(And I am sure Peter can help to find some reasonable workaround here for F12 target. In rawhide it will be solved properly by using the rules from lvm/dm package and switching to udev completely.)

Comment 2 David Zeuthen 2009-10-21 22:44:20 UTC
Yeah, I acknowledge this is a problem. I'll try to work on a fix soon. Thanks.

Comment 3 David Zeuthen 2009-10-24 02:18:19 UTC
I've committed a fix upstream such that we never touch a block device with a name "dm-*" on 'add', only on 'change'. This will be in F12 - I'll keep this bug open until then.

Comment 4 David Zeuthen 2009-10-24 02:28:00 UTC
(In reply to comment #0)
> 2) it must never scan and open internal devices, at least
> - temporary-cryptsetup
> - internal lvm devices like "pvmove"

And how do you suggest that we determine whether a given dm-* device is "internal"? Is there anything in sysfs to key off? Or do you expect to key off names?

From a 50,000 point of view, it's really problematic that device-mapper/LVM2 exports block devices that you don't want users or other programs to touch. It's akin to leaking implementation details into a public namespace and requires every program caring about block devices (normally disks and partitions) to special case support for LVM2/device-mapper, hide unwanted block devices and so on.

Comment 5 Milan Broz 2009-10-24 10:04:28 UTC
> And how do you suggest that we determine whether a given dm-* device is
> "internal"? Is there anything in sysfs to key off? Or do you expect to key off
> names?

I think that this is already solved by new udev rules, ask Peter Rajnoha for more details, but I think you already discussed that with him.

We are just solving workaround for F12 here. You can use DM device sybsystem which is part of UUID (available in sysfs), here for cryptsetup temporary device it is CRYPT-TEMP- prefix.
For udev rule see https://bugzilla.redhat.com/show_bug.cgi?id=526699#c5 - probably it can even do something simpler here.

(The name temporary-cryptsetup-* is just for older systems where the UUID is not yet set, F12 have it always.)

Please note that cryptsetup was written several years ago and the problem was introduced just recently by these udev rules which expects that DevKit can asynchronously scan any device in system. It is not correct assumption in this context.
I cannot retrospectively change design of kesylot processing because of this,
the implementation just create and destroy one temporary device per LUKS keyslot, it is not partition nor regular disk device. It is special block device which uses DM kernel crypt functionality to decrypt hashed volume key stored on special area of disk.
 
> From a 50,000 point of view, it's really problematic that device-mapper/LVM2
> exports block devices that you don't want users or other programs to touch.

From the other point of view is problematic that DevKit rules here duplicates udev functionality - and I hope it is solved already for future by moving all these scans to one place in unified dm udev rules.

Comment 6 David Zeuthen 2009-10-24 14:47:31 UTC
(In reply to comment #5)
> > From a 50,000 point of view, it's really problematic that device-mapper/LVM2
> > exports block devices that you don't want users or other programs to touch.
> 
> From the other point of view is problematic that DevKit rules here duplicates
> udev functionality - and I hope it is solved already for future by moving all
> these scans to one place in unified dm udev rules.  

Well, since device-mapper integration with udev is a new thing - not even in F12 as far as I can tell! - having DeviceKit-disks providing rules is the only way to do something sensible unless we want to cut features like LUKS integration with the desktop. Which of course we don't want to. Anyway, I'm looking forward to getting rid of my own rules.

My comment wasn't so much about udev rules per se. It was more a comment saying that it's not a good idea to export block devices that you don't want users or programs to see or touch. This behavior breaks a lot of programs (the various partition tools, one-offs automounters (e.g. simple udev rule), desktop integration etc.) and also makes it harder for users to figure out what's really going on. For example, admins/users _do_ expect that block devices are, well, block devices - e.g. some random access storage that can be probed, mounted, formatted and so on. So it is problematic that device-mapper/LVM2 breaks these assumptions by introducing block devices that, in a strict sense, really isn't - well, block devices.

Then again, it's too late to change all this.. and, FWIW, I'm not against adding special-case code to DeviceKit-disks to properly handle device-mapper / LVM2. Since GNOME and XFCE (and in the future other desktops such as KDE) uses DeviceKit-disks we only have to fix this problem once and for all (for the common use-cases anyway).

FWIW, I'm planning to add nice desktop UI in Palimpsest for LVM2 soon. Which should make LVM2 much more accessible to people.

Comment 7 Alasdair Kergon 2009-10-24 16:51:20 UTC
device-mapper pre-dates udev.  dm generalised the notion of a linux block device in certain ways.  Unfortunately udev didn't take those extensions into account in its original design, but we are at last very close to having a compatible interface between them which respects the requirements of both sides.

dm block devices are all block devices by definition.  The problems here stem from what we might term *private* block devices, intended only for specific use by a specific type of process.

The new udev rules Peter is writing attempt to allow such devices to be identified easily and isolated so that only the appropriate private udev rules are applied to them.

LVM2 sees this problem from both sides, because as well as creating and using private block devices, it also attempts to scan all block devices - currently still not integrated into udev, Peter's next project:-)  It too has to know how to skip 'private' devices, something it also fails to do 100% correctly as yet...

Comment 8 David Zeuthen 2009-10-24 17:07:07 UTC
(In reply to comment #7)
> dm block devices are all block devices by definition.  The problems here stem
> from what we might term *private* block devices, intended only for specific use
> by a specific type of process.

Is there any chance that we can have a simple sysfs attribute for block devices to mark that the device is private and should be skipped by 3rd party user space? It wouldn't have to be proprietary to device-mapper, could be part of the block layer proper.

Thanks,
David

Comment 9 Alasdair Kergon 2009-10-24 17:17:37 UTC
Currently 'global/private' is a userspace notion not a kernel (sysfs) one.  I'll give some thought to whether there's a reasonable way to introduce the notion into the kernel.

Comment 10 Peter Rajnoha 2009-10-29 10:38:56 UTC
> Is there any chance that we can have a simple sysfs attribute for block devices
> to mark that the device is private and should be skipped by 3rd party user
> space? It wouldn't have to be proprietary to device-mapper, could be part of
> the block layer proper.

David, would it be possible for you to check ENV{DM_UDEV_DISABLE_OTHER_RULES_FLAG}=="1"? This one is reserved for signalling the other (non-dm) rules not to process current event which is considered to be inappropriate/private. But this "flag support" was added in the newest lvm2 release only - v2.02.54.

(Well, we tried to use "last_rule" in our rules for this, so others don't need to do checks for themselves, but reading the udev news about removing "last_rule", we have to rely on others checking this flag instead from now on...)

Comment 11 James Laska 2009-10-30 17:18:08 UTC
This bug was reviewed during the F-12 Blocker meeting on 2009-10-30.  Unfortunately, we had a hard time determining the root cause and impact of this issue.  Can someone summarize what the problem is, whether workarounds exist, and the recommended solution?

Thanks!
James

Comment 12 David Zeuthen 2009-10-30 17:54:35 UTC
(In reply to comment #10)
> > Is there any chance that we can have a simple sysfs attribute for block devices
> > to mark that the device is private and should be skipped by 3rd party user
> > space? It wouldn't have to be proprietary to device-mapper, could be part of
> > the block layer proper.
> 
> David, would it be possible for you to check
> ENV{DM_UDEV_DISABLE_OTHER_RULES_FLAG}=="1"? This one is reserved for signalling
> the other (non-dm) rules not to process current event which is considered to be
> inappropriate/private. But this "flag support" was added in the newest lvm2
> release only - v2.02.54.
> 
> (Well, we tried to use "last_rule" in our rules for this, so others don't need
> to do checks for themselves, but reading the udev news about removing
> "last_rule", we have to rely on others checking this flag instead from now
> on...)  

Done in commit 	5a1e919e6692952b2e6384bb45ffb1d1c64af7b7.

(In reply to comment #11)
> This bug was reviewed during the F-12 Blocker meeting on 2009-10-30. 
> Unfortunately, we had a hard time determining the root cause and impact of this
> issue.  Can someone summarize what the problem is, whether workarounds exist,
> and the recommended solution?

This is fixed upstream with commits a95d351dcae957e36b1cd3c7c6c1a784de65dcf8 and 5a1e919e6692952b2e6384bb45ffb1d1c64af7b7 in DeviceKit-disks. There I'm closing this as UPSTREAM and making bug 528874 (which asks for a new DeviceKit-disks release) a blocker bug instead.

    David

Comment 13 Matthias Clasen 2009-10-30 22:22:43 UTC
Lets keep this open and on the blocker list until we get the fix in F12.

Comment 14 Matthias Clasen 2009-10-30 22:23:40 UTC
Sorry, misread your comment. You put another one on the blocker list...

Comment 15 Peter Rajnoha 2009-10-30 23:16:28 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > This bug was reviewed during the F-12 Blocker meeting on 2009-10-30. 
> > Unfortunately, we had a hard time determining the root cause and impact of this
> > issue.  Can someone summarize what the problem is, whether workarounds exist,
> > and the recommended solution?
> 
> This is fixed upstream with commits a95d351dcae957e36b1cd3c7c6c1a784de65dcf8
> and 5a1e919e6692952b2e6384bb45ffb1d1c64af7b7 in DeviceKit-disks.

Well, but ENV{DM_UDEV_DISABLE_OTHER_RULES_FLAG} will be available in udev-enabled libdevmapper with udev rules installed and that's going to be in F13 and later only, not in F12. So you still need to filter out those temporary-cryptsetup and pvmove dm devices manually in your rules for now (so that blkid, part-id etc. is not called).

Comment 16 Michal Schmidt 2009-11-02 08:34:10 UTC
There has been no new release of DeviceKit-disks yet, so I built it from git (latest HEAD 486854b8) to test it.

The error messages which appeared with 008 during boot ...:

 device-mapper: ioctl: unable to remove open device temporary-cryptsetup-420
 __ratelimit: 131 callbacks suppressed
 Buffer I/O error on device dm-13, logical block 64
 Buffer I/O error on device dm-13, logical block 65
 Buffer I/O error on device dm-13, logical block 66
 Buffer I/O error on device dm-13, logical block 67
 Buffer I/O error on device dm-13, logical block 68
 Buffer I/O error on device dm-13, logical block 69
 Buffer I/O error on device dm-13, logical block 70
 Buffer I/O error on device dm-13, logical block 71
 Buffer I/O error on device dm-13, logical block 72
 Buffer I/O error on device dm-13, logical block 73

... are now gone. I had to rebuild the initramfs first. I observe no problems now.

Comment 17 Adam Williamson 2009-11-02 23:39:34 UTC
There's a new build here:

http://koji.fedoraproject.org/koji/buildinfo?buildID=139475

tag request is here:

https://fedorahosted.org/rel-eng/ticket/3025

can you please test and confirm this build is enough to address this issue? Thanks!

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 18 Adam Williamson 2009-11-03 05:44:08 UTC
the build has been tagged into 2009-11-03 Rawhide; if you can confirm the build is good, we can close this report.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 19 Milan Broz 2009-11-03 09:27:11 UTC
*** Bug 532606 has been marked as a duplicate of this bug. ***

Comment 20 Peter Rajnoha 2009-11-03 14:34:31 UTC
OK, maybe a silly question and maybe I'm confused here, but what is the relation between DeviceKit-disks and dracut?

Hmm, and I've looked into the new package, but I don't see the filters for temporary-cryptsetup and pvmove dm devices there... We should filter these ones  - mainly for that blkid call that could cause problems. David, could you please add this? (that DM_UDEV_DISABLE_OTHER_RULES_FLAG will be in F13 only, but we still need to fix this for F12)

Comment 21 Michal Schmidt 2009-11-03 14:36:38 UTC
I am still seeing messages like:
device-mapper: ioctl: unable to remove open device temporary-cryptsetup-289
during boot, but they happen in the initramfs before the root is mounted (thus before the DeviceKit-disks rules come into effect, so it can't be DK-d's fault).
I suppose I am seeing bug 527056.

I have no other problems.

Comment 22 Michal Schmidt 2009-11-03 14:40:30 UTC
(correction of my comment #16)
> The error messages which appeared with 008 during boot ...:
> 
>  device-mapper: ioctl: unable to remove open device temporary-cryptsetup-420
>  __ratelimit: 131 callbacks suppressed
>  Buffer I/O error on device dm-13, logical block 64
>  ...
> 
> ... are now gone.

I was just lucky once. It is not perfectly deterministic, sometimes the errors are
 there and sometimes they aren't.

Comment 23 David Zeuthen 2009-11-03 14:45:18 UTC
(In reply to comment #20)
> OK, maybe a silly question and maybe I'm confused here, but what is the
> relation between DeviceKit-disks and dracut?

None - the former is for enumerating and performing operations on disks, the latter is for booting the system. Both use udev, blkid and other stuff but that's about it.

> Hmm, and I've looked into the new package, but I don't see the filters for
> temporary-cryptsetup and pvmove dm devices there... We should filter these ones
>  - mainly for that blkid call that could cause problems. David, could you
> please add this? (that DM_UDEV_DISABLE_OTHER_RULES_FLAG will be in F13 only,
> but we still need to fix this for F12)  

I think it would be a lot easier if you just set DM_UDEV_DISABLE_OTHER_RULES_FLAG in your F12 packages. That way you can finetune whatever needs to be ignored and we won't need to play any ping/pong back and forth ;-)

Otherwise please explain exactly how I'm supposed to ignore pvmove and temporary-cryptsetup nodes (by matching on dm/name? what if someone creates a volume with that actual name?) and I'll add the rules. Ideally with udev rules I can just copy/paste into the rules I use. Thanks!

Thanks,
David

Comment 24 Peter Rajnoha 2009-11-03 15:15:23 UTC
(In reply to comment #23)
> (In reply to comment #20)
> > OK, maybe a silly question and maybe I'm confused here, but what is the
> > relation between DeviceKit-disks and dracut?
> 
> None - the former is for enumerating and performing operations on disks, the
> latter is for booting the system. Both use udev, blkid and other stuff but
> that's about it.
> 

Exactly! That's what I thought too - I was just mystified a little with the comment #16 and I don't want this bug to be closed prematurely (looking at comment #17) :)

> I think it would be a lot easier if you just set
> DM_UDEV_DISABLE_OTHER_RULES_FLAG in your F12 packages. That way you can
> finetune whatever needs to be ignored and we won't need to play any ping/pong
> back and forth ;-)

Well, we don't have any udev rules within our package until udev_sync is compiled in and this is only in F13 version of lvm2/device-mapper package. So I have no place to set this for now...

> Otherwise please explain exactly how I'm supposed to ignore pvmove and
> temporary-cryptsetup nodes (by matching on dm/name? what if someone creates a
> volume with that actual name?) and I'll add the rules. Ideally with udev rules
> I can just copy/paste into the rules I use. Thanks!

I'd suggest moving that "device-mapper" part in your rules at the beginning of the rule file and just use that ENV{DKD_DM_NAME} and ENV{DKD_DM_UUID} to match the name/uuid. Don't be bothered about someone creating a volume with that actual name for now - it's not ideal, I know, but it's better than those errors shown when this filter is not applied at all.

So something like this should do its job (assuming there's DKD_DM_UUID as well):

ENV{DKD_DM_UUID}=="CRYPT-TEMP-?*", GOTO="devkit_disks_end"
ENV{DKD_DM_UUID}!="?*", ENV{DKD_DM_NAME}=="temporary-cryptsetup-?*", GOTO="devkit_disks_end"
ENV{DKD_DM_NAME}=="?*-pvmove[0-9]*", GOTO="devkit_disks_end"

Thanks

Comment 25 Peter Rajnoha 2009-11-03 15:20:51 UTC
(In reply to comment #21)
> I am still seeing messages like:
> device-mapper: ioctl: unable to remove open device temporary-cryptsetup-289
> during boot, but they happen in the initramfs before the root is mounted (thus
> before the DeviceKit-disks rules come into effect, so it can't be DK-d's
> fault).
> I suppose I am seeing bug 527056.

Yes, that's the case. Someone keeps the device open from udev rule while we're trying to close it in parallel - this is solved with that new udev_sync feature (these kind of problems are gone then). But unluckily, this won't be in F12. So we have to make these workarounds here and there for now in the rules. OK, I'll try to look which rule it is today :) Ah..

Comment 26 Adam Williamson 2009-11-03 16:02:34 UTC
Peter: "I don't want this bug to be closed prematurely (looking at comment #17) :)"

don't worry, the bug stays open as long as I'm seeing long comments I don't understand rather than short ones like 'yes, everything is fine!' :)

but tomorrow is pretty much the cut-off point for F12 final, so we need to know what the status of this is as far as blocking the release goes. basically I need to know once there's a koji build that, in your estimation, addresses enough of whatever problems this causes that it's OK to go ahead and release F12.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 27 David Zeuthen 2009-11-03 18:08:01 UTC
(In reply to comment #24)
> > I think it would be a lot easier if you just set
> > DM_UDEV_DISABLE_OTHER_RULES_FLAG in your F12 packages. That way you can
> > finetune whatever needs to be ignored and we won't need to play any ping/pong
> > back and forth ;-)
> 
> Well, we don't have any udev rules within our package until udev_sync is
> compiled in and this is only in F13 version of lvm2/device-mapper package. So I
> have no place to set this for now...

You can easily just include/generate the udev rule from the spec file.

> > Otherwise please explain exactly how I'm supposed to ignore pvmove and
> > temporary-cryptsetup nodes (by matching on dm/name? what if someone creates a
> > volume with that actual name?) and I'll add the rules. Ideally with udev rules
> > I can just copy/paste into the rules I use. Thanks!
> 
> I'd suggest moving that "device-mapper" part in your rules at the beginning of
> the rule file and just use that ENV{DKD_DM_NAME} and ENV{DKD_DM_UUID} to match
> the name/uuid. Don't be bothered about someone creating a volume with that
> actual name for now - it's not ideal, I know, but it's better than those errors
> shown when this filter is not applied at all.
> 
> So something like this should do its job (assuming there's DKD_DM_UUID as
> well):
> 
> ENV{DKD_DM_UUID}=="CRYPT-TEMP-?*", GOTO="devkit_disks_end"
> ENV{DKD_DM_UUID}!="?*", ENV{DKD_DM_NAME}=="temporary-cryptsetup-?*",
> GOTO="devkit_disks_end"
> ENV{DKD_DM_NAME}=="?*-pvmove[0-9]*", GOTO="devkit_disks_end"

OK, I've done this now:

http://cgit.freedesktop.org/DeviceKit/DeviceKit-disks/commit/?id=94287454c3931d6acc81d3e4dbe853a9ebcb8515

This change is in DeviceKit-disks-009-2.fc12:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1785602

Tag request:

https://fedorahosted.org/rel-eng/ticket/3054

Comment 28 Peter Rajnoha 2009-11-03 20:43:28 UTC
(In reply to comment #27)
> OK, I've done this now:
> 
> http://cgit.freedesktop.org/DeviceKit/DeviceKit-disks/commit/?id=94287454c3931d6acc81d3e4dbe853a9ebcb8515
> 
> This change is in DeviceKit-disks-009-2.fc12:
> 
> http://koji.fedoraproject.org/koji/taskinfo?taskID=1785602

Yes, this looks good. Thanks a lot!

(I would like to see all this stuff cleaned one day, but this is a transition so we have to get over it now. I hope all these things will be covered by our rules which are in F13 rawhide already. Just that "last_rule" removal from udev upstream has come into the way recently. Well, at least we can try to think about some general and more clever solution how to differentiate public/private devices now. Something more practical than setting env vars and checking them all around the rules...)

Comment 29 Bill Nottingham 2009-11-03 20:44:36 UTC
Given that, and that it will be in tomorrow's rawhide, closing.

Comment 30 Adam Williamson 2009-11-03 20:52:18 UTC
thanks a lot for all your work on this one, guys.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 31 David Zeuthen 2009-11-03 21:16:29 UTC
Actually, gah, this bugfix introduced a new bug - namely that we now attempt to automount all LVs. Which we really don't want to (at this point). Fortunately, this bug was catched by another distro, see

 https://bugs.freedesktop.org/show_bug.cgi?id=24885

So I've build DeviceKit-disks-009-3.fc12 with this fix:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1786613

Tag request here:

https://fedorahosted.org/rel-eng/ticket/3063

Sorry about that.

Comment 32 David Zeuthen 2009-11-03 21:19:11 UTC
(In reply to comment #28)
> (I would like to see all this stuff cleaned one day, but this is a transition
> so we have to get over it now. I hope all these things will be covered by our
> rules which are in F13 rawhide already. Just that "last_rule" removal from udev
> upstream has come into the way recently. Well, at least we can try to think
> about some general and more clever solution how to differentiate public/private
> devices now. Something more practical than setting env vars and checking them
> all around the rules...)  

Yeah - let's tackle this properly for F13! And thanks for all the help so far!


Note You need to log in before you can comment on or make changes to this bug.