Bug 819492 - GVfs apps should convey when eject/unmount takes a long time
GVfs apps should convey when eject/unmount takes a long time
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: gvfs (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomáš Bžatek
Fedora Extras Quality Assurance
AcceptedBlocker
:
Depends On:
Blocks: F17Blocker/F17FinalBlocker
  Show dependency treegraph
 
Reported: 2012-05-07 08:13 EDT by Kamil Páral
Modified: 2015-03-03 18:05 EST (History)
29 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-16 13:42:48 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
bug video demonstration (11.06 MB, video/webm)
2012-05-07 08:23 EDT, Kamil Páral
no flags Details
messages wrt to the whole video (1.72 MB, text/plain)
2012-05-07 09:06 EDT, Kamil Páral
no flags Details
Attempt at a gnome-shell proof-of-concept (4.00 KB, patch)
2012-05-14 16:37 EDT, Miloslav Trmač
no flags Details | Diff
Hack for shell (4.54 KB, patch)
2012-05-14 16:41 EDT, Adel Gadllah
no flags Details | Diff
Hack for shell (4.54 KB, patch)
2012-05-14 16:45 EDT, Adel Gadllah
no flags Details | Diff
quick update to the above (115.54 KB, patch)
2012-05-14 17:08 EDT, Bill Nottingham
no flags Details | Diff
Fixed up and tested version (4.54 KB, patch)
2012-05-14 17:10 EDT, Adel Gadllah
no flags Details | Diff
fixed patch (4.04 KB, patch)
2012-05-14 17:11 EDT, Bill Nottingham
no flags Details | Diff
Fixed up and tested version (4.41 KB, patch)
2012-05-14 17:11 EDT, Adel Gadllah
no flags Details | Diff
improved shell notifier (5.81 KB, patch)
2012-05-14 18:38 EDT, Cosimo Cecchi
no flags Details | Diff
improved shell notifier v2 (5.77 KB, patch)
2012-05-14 18:47 EDT, Cosimo Cecchi
no flags Details | Diff
nautilus patch (6.15 KB, patch)
2012-05-15 11:34 EDT, Tomáš Bžatek
no flags Details | Diff
notification after eject in nautilus (624.40 KB, image/png)
2012-05-15 18:09 EDT, Kamil Páral
no flags Details

  None (edit)
Description Kamil Páral 2012-05-07 08:13:31 EDT
Description of problem:
In Fedora 16 and earlier if you tried to unmount a USB flash drive and data were still being synced, you would receive a pop-up dialog "Copying data, don't eject the drive yet" (or similar). In Fedora 17 there is some bug and no such pop-up dialog appears. It seems that the drive can be ejected just fine, but when you do, you'll lose data.

I tested ext4 and NTFS formatted flash drives. For ext4 filesystems, incomplete files (not fully synced) usually are not displayed on the next mount, so you'll lose that file completely. For NTFS filesystems, incomplete files are usually displayed on the next mount, but they are corrupted (their contents differ from the original file). When using NTFS flash drives, there's a high probability you won't be able to mount the drive at all, I saw a error dialog about corrupted MFT entries several times and I had to format the whole disk again.

What does not work:
1. Unmounting flash drive from Nautilus. It is immediately unmounted, without waiting for data to be synced.
2. Unmounting from gnome-shell tray icon, ditto.

What works:
1. If you issue "sync" command after unmounting, and only then eject the device, no data is lost.
2. If you unmount manually by using "umount" command, no data is lost.

Version-Release number of selected component (if applicable):
nautilus-3.4.1-2.fc17.x86_64
gvfs-1.12.2-1.fc17.x86_64
gnome-shell-3.4.1-3.fc17.x86_64
http://www.smolts.org/client/show/pub_6528457b-60ef-4c7b-b6de-629feeab3d53

How reproducible:
always

Steps to Reproduce:
1. create large file, e.g. 300MB
2. copy to flash drive
3. unmount and eject
4. insert again and compare files
Comment 1 Kamil Páral 2012-05-07 08:23:01 EDT
Created attachment 582641 [details]
bug video demonstration

Either see included file or watch at http://www.youtube.com/watch?v=0uOTT1C5rdQ
Comment 2 Kamil Páral 2012-05-07 08:38:24 EDT
According to Tomas Bzatek this concerns all NTFS/ext* partitions (no matter whether thumb drive or external harddisk), but not FAT partitions (because they should be mounted in sync mode). I didn't try, but I don't think that really matters.

Proposing as F17 blocker. LiveCDs are often used for system backup/restore/troubleshooting purposes. If there is a high probability of corrupting data that were copied to USB devices, I don't believe we want to ship it as GOLD. It can't be fixed with an update.
Comment 3 Tomáš Bžatek 2012-05-07 08:54:59 EDT
This seems to me like udisks2 daemon forcing USB power down on the device before unmount operation (incl. fsync) is finished. Kamil, could you please attach full dmesg output here please?

Either way we're missing UI notification about umount progress too.
Comment 4 Kamil Páral 2012-05-07 09:06:20 EDT
Created attachment 582657 [details]
messages wrt to the whole video

Attaching messages. Please note I was able to do 'sync' after unmount in Nautilus, so it doesn't seem that the power supply was cut for that device.
Comment 5 Tomáš Bžatek 2012-05-07 09:14:52 EDT
Yeah, don't know what's really happening with sync running. This pattern is repeating however, I/O errors appearing always after device disconnect:

> May  7 13:44:09 localhost kernel: [ 1218.064041] usb 2-2: USB disconnect, device number 7
> May  7 13:44:10 localhost kernel: l block 51361
> May  7 13:44:10 localhost kernel: [ 1218.073798] Buffer I/O error on device sdb1, logical block 51362
> May  7 13:44:10 localhost kernel: [ 1218.073801] Buffer I/O error on device sdb1, logical block 51363
Comment 6 Adam Williamson 2012-05-07 11:48:11 EDT
Discussed at 2012-05-07 QA meeting, acting as a blocker review meeting. Accepted as a blocker per criterion "All known bugs that can cause corruption of user data must be fixed or documented at Common F17 bugs". Desktop team, what can be done about this? Thanks!



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 7 David Zeuthen 2012-05-07 11:56:55 EDT
(In reply to comment #3)
> This seems to me like udisks2 daemon forcing USB power down on the device
> before unmount operation (incl. fsync) is finished. Kamil, could you please
> attach full dmesg output here please?

udisks2 does no such thing.

> Either way we're missing UI notification about umount progress too.

Yes we are ... sure, it would be nice to have some UI notification about umount progress but the way it was implemented was really wacky (a session-wide process deciding to pop up windows) - it needs to be reimplemented in some saner way - ideally by the apps unmounting taking this responsibility.

I don't missing these notifications is a blocker in any way.
Comment 8 Adam Williamson 2012-05-07 12:07:02 EDT
it's not the fact of the missing notification, exactly. it's the fact that 'ejecting' a USB stick via GNOME does not do what a regular user would understand as ejection at all. the expected workflow from previous Fedora releases and other OSes (windows, android etc) is that when you 'eject' the stick, when that operation actually completes, it is safe to remove the stick. no-one should have to sit there and watch the light until it stops flashing, or know to go to a console and run 'sync'. that's absurd. there's various ways to handle it that aren't notifications - like the operation shouldn't complete until the data is actually synced - but the current status is clearly unacceptable.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 9 Adam Williamson 2012-05-07 12:34:04 EDT
Been talking to desktop team about this: their position is, broadly, that the notification popup is a nice added convenience that was coded in a bad way and removed because it was bad code. They don't feel its lack is a blocker issue:

<adamw> davidz: afaict right now, you either know to go to a console and type 'sync' and wait for it to return, know to watch the little flashing lights on your stick, or just wait some arbitrary amount of time before unplugging your stick and cross your fingers.
<davidz> nah, that's inventing stuff... most people just look at the LED on their drive
<davidz> and yank it when it's not flashing
<adamw> davidz: i...am not at all sure that's true.
<davidz> hell, people don't even unmount it
<davidz> that's a dream world
<davidz> I'm not disagreeing it's an inconvenience
<davidz> just saying it's not a block
<davidz> and that people who believe that most users are careful about dealing with usb sticks and are careful with unmount... are full of it

Matthias and David say the correct 'fix' for this would be in each application which actually handles disk mounting/unmounting - so at least gnome-shell, nautilus and the file chooser. gvfs 'does the right thing' in the sense that its unmount function does not actually return until data is synced, David says. So front end apps should either block until the unmount function returns, or display some kind of notification until it returns. But again, they don't consider the bug to a blocker and probably expect that to be F18 stuff.

So I'm moving this back to proposed blocker, and will try to get some firepower from devel, releng and FPL groups into the Friday meeting to give it a better review (the meeting today had only QA people present).



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 10 Matthias Clasen 2012-05-07 12:35:28 EDT
    It was a nice touch that we had this feature (the notification), and we should
    perhaps work on bringing it back in  a less wacky way.

    But it is entirely impractical to do that for f17 at this point, since it
    likely requires new api in the udisks2 service and new UI (thus translations).
Comment 11 David Zeuthen 2012-05-07 12:35:52 EDT
First of all, it's not a GVfs bug - GVfs actually does the right thing (it syncs) and it's relatively easy to convince yourself of this, see [1].

So this means that if you want the behavior described in comment 8 you will need to patch all apps calling g_mount_unmount() ... this includes (at least)

 - Shell (gnome-shell)
 - GTK's File Chooser (GtkFileChooser)
 - Files (nautilus).

Before you do this, you of course need to design the feature - what we had pre-f17 with http://git.gnome.org/browse/gnome-disk-utility/tree/src/notification/notification-main.c?h=gnome-3-2#n273 was a really bad hack which is completely not suitable for GNOME 3 where we do system-modal dialogs in a different way. That's why it was yanked.

So there is some work left here.

And this work starts with design.

(and, once you actually do the design and start thinking about the problem, you may find that this socalled feature is not needed at all - for example, it may be enough to simply observe that most devices has a LED that is blinking while the device is busy.... of course, now some people say that not all devices have a LED or it's too dim or whatever...)

--

The other point is that I think this is not a blocker and you do. I don't want to get into this kind of discussion so please have it elsewhere. But do note that there is no way to bring the pre-f17 dialog back. None whatsoever.

[1] : as you can see running test.sh from [2] there's a 14 second delay before g_mount_unmount() returns (it's a crappy USB stick so in line with single-digits MB/s performance)

$ ./test.sh
mounting device at Mon May  7 12:21:02 EDT 2012
Mounted /dev/sdb1 at /run/media/davidz/foo
clearing file at Mon May  7 12:21:02 EDT 2012
starting dd a 100MB file at Mon May  7 12:21:02 EDT 2012
100+0 records in
100+0 records out
100000000 bytes (100 MB) copied, 0.0791256 s, 1.3 GB/s
unmounting at Mon May  7 12:21:02 EDT 2012
done unmounting at Mon May  7 12:21:16 EDT 2012


[2] : test.sh .. you may need to adjust to make it work on your own syste, / login

#!/bin/bash

set -e

echo "mounting device at `date`"
gvfs-mount -d /dev/sdb1
echo "clearing file at `date`"
rm -f /run/media/$USER/foo/bar
echo "starting dd a 100MB file at `date`"
dd if=/dev/zero of=/run/media/$USER/foo/bar bs=1000000 count=100
echo "unmounting at `date`"
gvfs-mount -u /run/media/$USER/foo/bar
echo "done unmounting at `date`"
Comment 12 Kalev Lember 2012-05-07 12:44:06 EDT
(In reply to comment #5)
> Yeah, don't know what's really happening with sync running. This pattern is
> repeating however, I/O errors appearing always after device disconnect:
> 
> > May  7 13:44:09 localhost kernel: [ 1218.064041] usb 2-2: USB disconnect, device number 7
> > May  7 13:44:10 localhost kernel: l block 51361
> > May  7 13:44:10 localhost kernel: [ 1218.073798] Buffer I/O error on device sdb1, logical block 51362
> > May  7 13:44:10 localhost kernel: [ 1218.073801] Buffer I/O error on device sdb1, logical block 51363

This is likely kparal physically yanking out the usb device, and not some software initiated power down.
Comment 13 Adam Williamson 2012-05-07 13:32:03 EDT
Upstream has a similar bug to this:

https://bugzilla.gnome.org/show_bug.cgi?id=670452

i.e. it claims that gvfs is broken, but it isn't. Upstream also has a 'correct' bug for Nautilus:

https://bugzilla.gnome.org/show_bug.cgi?id=619665

and I filed one for Shell:

https://bugzilla.gnome.org/show_bug.cgi?id=675627

so we have the two key apps covered now.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 14 David Zeuthen 2012-05-07 14:03:33 EDT
Changing summary to be more accurate and reassigning to the GVfs maintainer to boot ... could close this as NOTABUG since there actually is no bug in GVfs (per comment 11) and it's all in the apps using it. But keeping it open since we can use it as a tracker bug. Thanks.
Comment 15 Kamil Páral 2012-05-09 04:42:14 EDT
Just to clarify:

1. I have 3 flash drives at home, *none* of them has any LED indicator.

2. The flash drive I reported this bug on (see video) has about 3-4MB/s write speed, and about 800MB is cached instantly when copying a large file. That is roughly 4 minutes of additional write operations after the moment the copying has seemingly finished. This is confirmed with my hands-on experience.

3. Believing that people would wait for some magic timeout before they yank the drive out is completely out of question. I (as a power user on top of that) am a good example that it wouldn't work, because I have seen my data corrupted several times until I found that bug. Maybe you haven't ever used flash drives before, but the real-world workflow is: "you copy the data onto the flash drive -> you click unmount icon or 'eject' menu item -> you yank the drive out". This is the way it works in all operating systems. And it has also worked this way in Fedora.

If you ever intended to get a terrible reputation for losing your users' data, removing notifications is a great way to go. If anyone is really serious about this (I can't believe that), I want to see this blessed by FESCo, this is a way too important regression.
Comment 16 teppot 2012-05-09 12:56:39 EDT
Could we mount removable media synchronously?
Comment 17 Bojan Smojver 2012-05-09 19:14:37 EDT
(In reply to comment #11)
> First of all, it's not a GVfs bug - GVfs actually does the right thing (it
> syncs) and it's relatively easy to convince yourself of this, see [1].

Can't GVfs fire off a general notification when any unmount is complete?
Comment 18 Adam Williamson 2012-05-10 23:04:30 EDT
I really don't know which way to vote on this one. I can kinda see both perspectives on it. Count me +/-0.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 19 Samuel Sieb 2012-05-11 00:37:01 EDT
This is going to cause major dataloss issues.  Every other OS that I know of provides an indication that you shouldn't pull the USB drive out while it finishes writing.
Comment 20 Kamil Páral 2012-05-11 03:44:17 EDT
I won't be around during today's blocker bug meeting. My vote is clear +1 blocker.

It is not really important which solution we implement, but we need _some_ way to tell users "don't eject yet". I know, there's not much time. But LiveCDs won't get updated and they are used for system rescue and similar purposes.

If consensus is not reached I'll propose this topic for the next FESCo meeting.
Comment 21 Adam Williamson 2012-05-11 20:53:55 EDT
Discussed at the 2012-05-11 blocker review meeting: http://meetbot.fedoraproject.org/fedora-bugzappers/2012-05-11/f17-final-blocker-review-meeting-5.2012-05-11-17.04.html . This was accepted as a blocker, conditionally: it was accepted that a 'clean' fix is unlikely to be possible on the F17 timescale, but the meeting produced a clear consensus that there needs to be _some_ communication to the user of when it's safe to disconnect storage media. Also note that the issue may be escalated to FESCo.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 22 Ankur Sinha (FranciscoD) 2012-05-13 08:41:10 EDT
I saw this too. I actually lost data, because I expected the system to only unmount my device after the data operation was done (even if it didn't pop up a dialogue to tell me about it). I was surprised to see my data missing on the drive. 

Blocker +1 for me. It does cause data loss, and is therefore a loss of usability. Imagine copying something and reaching a meeting only to realize that the copy hadn't completed (and how would you know!?)? 

I recently bought two nice, steel HP Pen drives, and none of them have LED indicators (like Kamil already pointed out). I just go for a walk and give it 10 minutes to finish, but I don't expect everyone to start following that :)

Ankur
Comment 23 Kamil Páral 2012-05-14 08:16:42 EDT
There were some voices asking me to propose this issue to FESCo in all cases, because QA viewpoint seems to collide with developers viewpoint. So I did:
https://fedorahosted.org/fesco/ticket/850
Comment 24 Tomáš Bžatek 2012-05-14 10:21:23 EDT
Quick and dirty proposal for Nautilus:

Add a timeout (say 1.0 - 1.5 sec) right after g_mount_eject_with_operation() or g_mount_unmount_with_operation() is called. This is taken place either in libnautilus-private/nautilus-file-operations.c:do_unmount() and src/nautilus-places-sidebar.c:do_eject(). The timeout callback would display a notification that something is going on, either by opening a separate window or using libnotify. Once the g_mount_xxx_with_operation() is done, either destroy the source or the visible window/notification.

At this point we depend heavily on the particular gvfs volume monitor to spawn the callback actually when the mount is safely unmounted.


This way we would have some kind of notification that would satisfy the missing UI feedback of ongoing unmount operation going on. It's not meant to be included upstream in any way, this is just for Fedora 17 release.
Comment 25 Matthias Clasen 2012-05-14 14:12:47 EDT
Tomas, would this help much ?
Considering we're not running nautilus by default anymore...
Comment 26 Adel Gadllah 2012-05-14 14:20:44 EDT
(In reply to comment #16)
> Could we mount removable media synchronously?

NO. This would suck horribly.
Comment 27 Matthias Clasen 2012-05-14 14:26:07 EDT
(In reply to comment #26)
> (In reply to comment #16)
> > Could we mount removable media synchronously?
> 
> NO. This would suck horribly.

Sorry, this has been declared a blocker, so we have to go for the sucky fix.
Tomas, can you look at a patch for sync mounting ?
Comment 28 Miloslav Trmač 2012-05-14 14:29:36 EDT
(In reply to comment #27)
> Sorry, this has been declared a blocker, so we have to go for the sucky fix.
> Tomas, can you look at a patch for sync mounting ?

That doesn't necessarily mean "the most sucky fix possible".
Comment 29 Miloslav Trmač 2012-05-14 14:32:57 EDT
(In reply to comment #28)
> (In reply to comment #27)
> > Sorry, this has been declared a blocker, so we have to go for the sucky fix.
> > Tomas, can you look at a patch for sync mounting ?
> 
> That doesn't necessarily mean "the most sucky fix possible".

And, to be specific, mounting "sync" does not mean that any time a medium is removed, its contents will be the same as if it were cleanly unmounted - and it doesn't mean so even if it were removed in a "quiet" period.
Comment 30 Kamil Páral 2012-05-14 14:41:45 EDT
FESCo confirmed they also see this issue as a blocker to Fedora 17 release:
http://meetbot.fedoraproject.org/fedora-meeting/2012-05-14/fesco.2012-05-14-17.00.log.html (search for "#850")

Please note that mounting removable drives in sync mode is one possible way to explore, but if better short-term fixes are found then they are of course also a way to go.
Comment 31 Kamil Páral 2012-05-14 16:04:27 EDT
David Zeuthen posted his opinion on sync mounting here:
https://fedorahosted.org/fesco/ticket/850#comment:4

I propose to continue the discussion here, so it's in a single place only.
Comment 32 Miloslav Trmač 2012-05-14 16:37:31 EDT
Created attachment 584469 [details]
Attempt at a gnome-shell proof-of-concept

This extremely ugly patch demonstrates the place that apparently needs to be extended in gnome-shell.  It just puts a label (no background, so pretty much invisible) at (100, 100) - this needs to be replaced by a reasonable dialog or notification, perhaps importing and reusing the translations from the F16 dialog.

Take with a grain of salt - my understanding of the deep library stack and generated D-Bus wrappers isn't great.

I can't give this much more time, I'm afraid - perhaps this will help someone a bit.
Comment 33 Adel Gadllah 2012-05-14 16:41:57 EDT
Created attachment 584470 [details]
Hack for shell

Here is a quick hack for unmounting from withing gnome-shell (untested) ... Matthias suggested that if we want to add something like that it should be consistent (i.e both hacks behave the same).
Comment 34 Adel Gadllah 2012-05-14 16:45:04 EDT
Created attachment 584471 [details]
Hack for shell
Comment 36 Adel Gadllah 2012-05-14 17:10:50 EDT
Created attachment 584477 [details]
Fixed up and tested version
Comment 37 Bill Nottingham 2012-05-14 17:11:07 EDT
Created attachment 584478 [details]
fixed patch
Comment 38 Adel Gadllah 2012-05-14 17:11:53 EDT
Created attachment 584479 [details]
Fixed up and tested version
Comment 39 Adam Williamson 2012-05-14 17:32:14 EDT
I can think of at least two other just GNOME apps that mount/unmount stuff, I think - Shotwell and Rhythmbox. Not to mention all the non-GNOME apps - Calibre, for e.g. This kinda feels like something which should be abstracted/centralized somehow. GTK+?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 40 David Zeuthen 2012-05-14 18:04:13 EDT
Was debugging this with Cosimo today and cherry-picked the following patches

http://git.gnome.org/browse/gvfs/commit/?h=gnome-3-4&id=052010c7ca25217fd9f3a4a9d479d98ef5f57ae5
http://git.gnome.org/browse/gvfs/commit/?h=gnome-3-4&id=0a47bf098e4d33c6284b9f2ef47c175c6503b932

to GVfs' gnome-3-4 branch (so they will be in the next GVfs release for 3.4). The first patch is needed to detect if the filesystem is busy (umount(8) changed its output...). The second patch is needed in case it takes more than 25 secs to run umount(8)... which is totally can given how slow USB sticks are and how big the kernel-side cache is.

Additionally, udisks2 needs a similar patch as the first one... this is already in rawhide's udisks-1.97-1.fc18 packages, but I've built it into F17 here

 http://koji.fedoraproject.org/koji/taskinfo?taskID=4076910

for udisks-1.94-3.fc17. I'll file an update too.
Comment 41 David Zeuthen 2012-05-14 18:07:26 EDT
(In reply to comment #40)
> which is totally can given how slow USB sticks are
> and how big the kernel-side cache is.

I meant to say "totally possible", not "totally can". Sorry for the confusion.
Comment 42 Fedora Update System 2012-05-14 18:09:28 EDT
udisks2-1.94.0-3.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/udisks2-1.94.0-3.fc17
Comment 43 David Zeuthen 2012-05-14 18:19:33 EDT
Actually you want http://koji.fedoraproject.org/koji/taskinfo?taskID=4076920 since I forgot to add a %patch1 directive to the spec file...
Comment 44 Fedora Update System 2012-05-14 18:21:34 EDT
udisks2-1.94.0-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/udisks2-1.94.0-4.fc17
Comment 45 Cosimo Cecchi 2012-05-14 18:38:51 EDT
Created attachment 584489 [details]
improved shell notifier

This is an improved shell notifier based on Adel's:
- do not show the notification when the unmount operation fails because some file descriptors are open on the device
- make the notification transient + urgent for the duration of the data flushing, and let it time out with a "You can now unplug the device" when flushing finishes
- improve notification message
- make the notification use the same icon as the removable devices shell menu
Comment 46 Cosimo Cecchi 2012-05-14 18:47:15 EDT
Created attachment 584491 [details]
improved shell notifier v2

Remove a leftover debug comment
Comment 47 Adam Williamson 2012-05-14 19:14:48 EDT
it would be best to gather *all* the planned changes to fully address this issue and submit them as a single update. AIUI the udisks2 update submitted above does not really fully address this bug on its own.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 48 Kamil Páral 2012-05-15 07:40:19 EDT
Can somebody summarize what is going to be patched in the end? Currently I see patches for:

gvfs        (no new build yet)
udisks2     (new build available)
gnome-shell (no new build yet)

Do other applications (nautilus, shotwell, rhythmbox) require patches as well, or will the above patches fix it automatically for them?

As Adam said, please bundle all the patches to the udisks2 update David already posted, thanks.
Comment 49 Tomáš Bžatek 2012-05-15 11:34:53 EDT
Created attachment 584696 [details]
nautilus patch

Nautilus patch, first try. Attempts to mimics shell mounter behaviour with an addition of timeout for devices that sync quickly.

This patches sidebar only, there are few more eject/unmount op occurrences in nautilus-vfs-file.c and nautilus-file-operations.c sources.
Comment 50 David Zeuthen 2012-05-15 12:06:38 EDT
(In reply to comment #39)
> I can think of at least two other just GNOME apps that mount/unmount stuff, I
> think - Shotwell and Rhythmbox. Not to mention all the non-GNOME apps -
> Calibre, for e.g. This kinda feels like something which should be
> abstracted/centralized somehow. GTK+?

Yes, for GNOME 3.6 and F18 we are considering a solution that will Just Work(tm) for any app using GtkMountOperation. See

 https://bugzilla.gnome.org/show_bug.cgi?id=676111

We obviously can't do this for GNOME 3.4 and F17 since it involves new API.
Comment 51 Tomáš Bžatek 2012-05-15 14:25:36 EDT
Used slightly modified patch, duplicating the notification in nautilus-file-operations.c. Didn't go for patching nautilus-vfs-file.c as long as the unmount/eject functions can be called on mounts not representing physical devices (i.e. pure gvfs mounts) and the messages could be misleading.

(In reply to comment #47)
> it would be best to gather *all* the planned changes to fully address this
> issue and submit them as a single update. AIUI the udisks2 update submitted
> above does not really fully address this bug on its own.

Built nautilus-3.4.2-2.fc17 and gvfs-1.12.3-1.fc17, didn't file an update, please pick them up if want to submit a single update for all of them. I would include them in mass gnome 3.4.2 update otherwise.

Please also test to see if the solution provided is sufficient to resolve the blocker.
Comment 52 Adam Williamson 2012-05-15 14:32:41 EDT
I believe we also need a Shell build? thanks!



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 53 Cosimo Cecchi 2012-05-15 15:00:29 EDT
I updated again the shell patch to better handle some error conditions, and multiple potentially blocking eject operations simultaneously, and sent this upstream here [1].

[1] https://bugzilla.gnome.org/show_bug.cgi?id=676125
Comment 54 Kamil Páral 2012-05-15 15:15:00 EDT
I updated my computer with these builds:
udisks2-1.94.0-4.fc17
nautilus-3.4.2-2.fc17
gvfs-1.12.3-1.fc17

and tried to reproduce the problem in Nautilus. It still behaves the same. After I click eject in Nautilus, the 'eject' icon disappears, but nothing else happens - no dialog is thrown, no spinner is shown. If I unplug the device, my files are corrupted. If I wait long enough, the device disappears from the Nautilus side bar and only then it is safe to remove the device - but that's completely non-intuitive (it may have worked that way even before, I don't know).

Is that a problem in the Nautilus build, or am I just missing the new Gnome-shell build for the fix to manifest properly?
Comment 55 Adel Gadllah 2012-05-15 16:04:37 EDT
(In reply to comment #54)
> I updated my computer with these builds:
> udisks2-1.94.0-4.fc17
> nautilus-3.4.2-2.fc17
> gvfs-1.12.3-1.fc17
> 
> and tried to reproduce the problem in Nautilus. It still behaves the same.
> After I click eject in Nautilus, the 'eject' icon disappears, but nothing else
> happens - no dialog is thrown, no spinner is shown. If I unplug the device, my
> files are corrupted. If I wait long enough, the device disappears from the
> Nautilus side bar and only then it is safe to remove the device - but that's
> completely non-intuitive (it may have worked that way even before, I don't
> know).
> 
> Is that a problem in the Nautilus build, or am I just missing the new
> Gnome-shell build for the fix to manifest properly?

No, the shell patches are only needed for cases where you unmount from within the shell.

Do you have notifications enabled? (Switch in the usermenu).
Comment 56 Cosimo Cecchi 2012-05-15 16:24:53 EDT
FYI I now updated the Nautilus patch to match the latest Shell patch behavior wrt. strings and handling of multiple volumes, and posted it upstream here [1].
I am not particularly picky about which patch ends up in the final spin, but can provide another Nautilus build right away with this patch if you want.

[1] https://bugzilla.gnome.org/show_bug.cgi?id=619665#c13
Comment 57 Tomáš Bžatek 2012-05-15 16:55:18 EDT
(In reply to comment #54)
> nautilus-3.4.2-2.fc17

Uh oh, my fault, a leftover in the spec file from testing. Please try with nautilus-3.4.2-3.fc17

Reminds me... comment 43 :-)
Comment 58 Kamil Páral 2012-05-15 18:09:37 EDT
Created attachment 584797 [details]
notification after eject in nautilus

nautilus-3.4.2-3.fc17.x86_64 displays notification, see attached image. The notification seems to persist until the data is synced or until you click it. In both cases, especially when you click it, it is somewhat hard to understand how you'll know when you can eject the device. Usually notifications are used to alert user something just happened, not that something _is happening_, so it's slightly confusing. Is it possible to add a spinner or an indefinite progress bar inside that notification? That would create a much better feeling "please wait, work in progress".

After the data is synced, the notification text is changed to "You can now eject the device", but it flashes very briefly (0.5 second) and disappears.

It's definitely an improvement over the previous state, thanks. If we can do some minor fixes in the remaining short time, my ideas are:
1. add spinner or indefinite progress bar to the notification
2. make "you can now eject the device" notification stay longer, maybe until clicked or device ejected if that's possible, otherwise at least 5 seconds
3. is changing notification into a pop-up dialog completely out of question? the notification has some disadvantages - it can be closed with a click (which confuses user what to do next) and it blocks other notifications (data syncing may last several minutes).
4. re-use translations from the old F16 dialog

But even the current state looks good enough to close this bug, I think.

We still need the gnome-shell build, though.
Comment 59 Adam Williamson 2012-05-15 18:36:23 EDT
especially it would be good if we could prevent the notification from being closed if clicked on.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 60 Cosimo Cecchi 2012-05-15 21:58:11 EDT
(In reply to comment #58)

> After the data is synced, the notification text is changed to "You can now
> eject the device", but it flashes very briefly (0.5 second) and disappears.

> 2. make "you can now eject the device" notification stay longer, maybe until
> clicked or device ejected if that's possible, otherwise at least 5 seconds

This was fixed in my version of the Nautilus patch (which is in nautilus-3.4.2-4.fc17)
Comment 61 Adam Williamson 2012-05-15 22:03:01 EDT
Can people please test the update that fixes this and file karma as a matter of urgency? We need karma to pull this into the RC compose. Thanks!

https://admin.fedoraproject.org/updates/FEDORA-2012-7859/nautilus-3.4.2-4.fc17,gnome-shell-3.4.1-4.fc17,gvfs-1.12.3-1.fc17,udisks2-1.94.0-4.fc17

It now has all four necessary builds, and I've tested it myself and found it seems to work.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 62 Adam Williamson 2012-05-16 01:54:28 EDT
Setting ON_QA as the fix is in TC6 and updates_testing. Please test and karma.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 63 Fedora Update System 2012-05-16 07:52:02 EDT
nautilus-3.4.2-4.fc17, udisks2-1.94.0-4.fc17, gvfs-1.12.3-1.fc17, gnome-shell-3.4.1-4.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 64 David Zeuthen 2012-05-16 10:10:27 EDT
Btw, Cosimo pointed out this morning that the root problem of all this is actually that the app is Doing It Wrong(tm) - e.g. if the app copying stuff onto the device was things like fsyncdata(2) at strategic times (e.g. without sacrificing performance), the user would see a steady progress bar.

While we talked about this, I remembered that I actually filed this for Nautilus years ago, see

 https://bugzilla.gnome.org/show_bug.cgi?id=579085

API-wise, the fix could be as simple as introducing a new GFileCopyFlags flag to request that something like the above is done. Then Nautilus, and other GIO applications, can use this flag when copying.
Comment 65 Miloslav Trmač 2012-05-16 10:29:22 EDT
(In reply to comment #64)
> Btw, Cosimo pointed out this morning that the root problem of all this is
> actually that the app is Doing It Wrong(tm) - e.g. if the app copying stuff
> onto the device was things like fsyncdata(2) at strategic times (e.g. without
> sacrificing performance), the user would see a steady progress bar.

No, reliability of storing data is pretty much a system-wide property and asking every application to take care of it doesn't make sense.  Who will ever replace all of the write() and fwrite() calls in those tens of millions lines of code?


Yes, we gain performance by sacrificing ideal reliability of storing data when we use write-back caching at all (and that's pretty much from the oldest versions of UNIX) - but then it's up to the administrative tools, in this case the tools provided to unmount volumes or shut down the system, to build a friendly and reliable user interface that allows both programmers not to worry about this all the time, and users to understand how to use the system correctly.
Comment 66 David Zeuthen 2012-05-16 11:00:13 EDT
(In reply to comment #65)
> (In reply to comment #64)
> > Btw, Cosimo pointed out this morning that the root problem of all this is
> > actually that the app is Doing It Wrong(tm) - e.g. if the app copying stuff
> > onto the device was things like fsyncdata(2) at strategic times (e.g. without
> > sacrificing performance), the user would see a steady progress bar.
> 
> No, reliability of storing data is pretty much a system-wide property and
> asking every application to take care of it doesn't make sense.  Who will ever
> replace all of the write() and fwrite() calls in those tens of millions lines
> of code?
> 
> 
> Yes, we gain performance by sacrificing ideal reliability of storing data when
> we use write-back caching at all (and that's pretty much from the oldest
> versions of UNIX) - but then it's up to the administrative tools, in this case
> the tools provided to unmount volumes or shut down the system, to build a
> friendly and reliable user interface that allows both programmers not to worry
> about this all the time, and users to understand how to use the system
> correctly.

The point is this: if a "dumb" app is leading you to think that it's done copying (it's not, your data is not yet on stable platters), then, yes, we should show the "waiting to sync, don't unplug" notification if you eject/umount it before. In fact, since a lot of people don't even bother to umount/eject, this is a lot better since they won't yank the drive until the copy progress window is gone.

I'm just saying "don't write dumb apps" and that dumb apps are typically the root problem of symptoms like the one reported here. Of course, that's horribly naive but _at least_ we can try to aim for something better.

That's all.
Comment 67 Adam Williamson 2012-05-16 13:05:42 EDT
We have solid testing that at least nautilus and shell are fixed in TC6, and they're the main ones, we think.

Should we leave this bug open (but drop its 17 Blocker status) to track the implementation of the proper fix in GNOME 3.6 for Rawhide? Or should we just close it?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 68 Cosimo Cecchi 2012-05-16 13:31:16 EDT
(In reply to comment #67)

> Should we leave this bug open (but drop its 17 Blocker status) to track the
> implementation of the proper fix in GNOME 3.6 for Rawhide? Or should we just
> close it?

There's a number of bug reports are already open upstream for 3.6. and this bug already has 67 comments and 28 people in the CC list, so I'd say close it.
Comment 69 Adam Williamson 2012-05-16 13:42:48 EDT
OK.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 70 Fedora Update System 2012-05-22 08:33:30 EDT
udisks2-1.94.0-5.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/udisks2-1.94.0-5.fc17
Comment 71 Fedora Update System 2012-05-29 12:23:08 EDT
udisks2-1.94.0-5.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 72 matmenu8 2012-09-24 06:37:23 EDT
As mentionned here : https://bugzilla.redhat.com/show_bug.cgi?id=838237
I'm still experiencing data losses on USB Drives. I never had an answer, is that really the actual policy of Fedora to let it's userbase with dataloss and tell them it's "normal behaviour" ? 
If so, please at least ell people on boot time or on the main page before they download that data safety for people who don't know/want to use the sync command, is not a priority anymore for the devs.
I found this bug report by speaking with other people with the same problem on a french forum.
Comment 73 Kamil Páral 2012-09-24 06:47:20 EDT
I no longer see data loss on USB flash drives, but I really did see some strange behavior on USB external HDD (disk was kept displayed in Nautilus, no notification appeared wrt unplugging, I received errors in dmesg after disconnect).

matmenu8, can you please create a new bug, include all the details, and link it here?

Cosimo, David or Matthias, can you please link here upstream GNOME bugs that are related to the new GNOME 3.6 functionality? And can you please describe what works differently in there when compared to 3.4?

Thanks.
Comment 74 matmenu8 2012-09-24 07:13:04 EDT
I just came back from holyday, i just tried to safely remove my USB HDD, the option is gone in gnome shell/nautilus/disk utility.
I googled and came on another hot topic with the hardcore terminal lover against the user who only wants an easy way to remove usb drives and the ecologist who do not want to waste elecricity by powering down it's hdd when not used and the electrician who speaks about the danger of removing a device when still powered.
AND THE WINNER IS : the hardcore terminal lover.
Sorry i won't use time to fight on another regression or wait 2 month without any answer on such a crucial point. I go use something else that which i can trust for my data.

Good luck with that for those who stay. I'll keep an eye if the gnome/Fedora devs hear the community again some days. What a shame.
Comment 75 Kamil Páral 2012-12-07 10:02:21 EST
matmenu8, I have found the problem and reported it as bug 885133. Your experience is welcome (the type and brands of affected devices, for example). If you can, please provide the information in that new bug. Thank you.

Note You need to log in before you can comment on or make changes to this bug.