Bug 1225957
Summary: | Regression: GFileMonitor doesn't react to "mv some-file watched-file" | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Marius Vollmer <mvollmer> | ||||||
Component: | glib2 | Assignee: | Matthias Clasen <mclasen> | ||||||
Status: | CLOSED NEXTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 23 | CC: | allison.karlitskaya, awilliam, bugzilla, herrold, klember, mclasen, robatino, sgallagh, stefw, walters | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
URL: | https://bugzilla.gnome.org/show_bug.cgi?id=751358 | ||||||||
Whiteboard: | AcceptedBlocker | ||||||||
Fixed In Version: | 1.45.4-1.fc23 | Doc Type: | Bug Fix | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2015-08-25 16:06:12 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 1170819 | ||||||||
Attachments: |
|
Description
Marius Vollmer
2015-05-28 14:24:17 UTC
This is pretty bad, I think. Could I have some reaction? > Apologies in advance if this is a bogus bug report. I didn't carefully investigate yet.
I did investigate now, and the bug seems very real.
I couldn't see any difference in inotify behavior, but GFileMonitor fails with the attached test program.
Created attachment 1041869 [details]
test program
gcc -o t-GFileMonitor t-GFileMonitor.c $(pkg-config --libs --cflags glib-2.0 gio-2.0)
Created attachment 1041870 [details]
test script
On Fedora 22: [root@f22 work]# ./BUG changed 3 changed 0 changed 1 changed 3 On Rawhide: [root@fraw work]# ./BUG changed 3 changed 0 changed 1 This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle. Changing version to '23'. (As we did not run this process for some time, it could affect also pre-Fedora 23 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23 Ryan Lortie did lots of refactoring of the GFileMonitor code in GLib earlier this year. Perhaps this is related? The following commits landed in 2.45.1 commit d682df186e9cca2b8db0e921450a4db31cf0c467 Author: Ryan Lortie <desrt> Date: Sat Jan 17 11:11:22 2015 -0500 file monitors: rewrite FAM file monitor Completely rewrite the FAM file monitor. Major changes: - now runs in the worker thread - dispatches events in a threadsafe way via GFileMonitorSource - uses unix fd source instead of a GIOChannel - is now simple enough to fit into one short file commit 21ab660cf8d2622917eceaa24fbd07d81ea949af Author: Ryan Lortie <desrt> Date: Fri Jan 16 20:25:38 2015 -0500 fen: remove Solaris file monitor support This code is unmaintained and we have no way to port it to the new file monitoring API. commit 641b98ce6b7f58d32001002881f049be4680429b Author: Ryan Lortie <desrt> Date: Fri Jan 16 18:51:34 2015 -0500 kqueue backend: port to new GLocalFileMonitor API This is the bare minimal effort. This seems not to crash immediately, but it definitely needs some better testing. The backend is not in good shape. It could use some serious work. commit 19522424b15ff4b0e2be6dc453183e836953d20c Author: Ryan Lortie <desrt> Date: Fri Jan 16 16:41:21 2015 -0500 GPollFileMonitor: use thread default main context Attach the GPollFileMonitor to the thread default main context instead of the global default. This matches the behaviour of the other file monitors. commit c5d5e2916bc2ce81e4882167f1439570f0ac178e Author: Ryan Lortie <desrt> Date: Thu Jan 15 16:38:22 2015 -0500 inotify: implement "boredom" algorithm Use the "interesting" value from g_file_monitor_source_handle_event() to decide if we're currently being flooded by a stream of boring events. The main case here is when one or more files is being written to and the change events are all being rate-limited in the GFileMonitor frontends. In that case, we become "bored" with the event stream and add a backoff timeout. In the case that it is exactly one large file being written (which is the common case) then leaving the event in the queue also lets the kernel perform merging on it, so when we wake up, we will only see the one event. Even in the case that the kernel is unable to perform merging, the context switch overhead will be vastly reduced. In testing, this cuts down on the number of wake ups during a large file copy, by a couple orders of magnitude (ie: less than 1% of the number of wake ups). commit 9adf948a2b2215814bbc6b10e827349d05f2c883 Author: Ryan Lortie <desrt> Date: Thu Jan 15 15:39:43 2015 -0500 GFileMonitorSource: return "interesting" value Return an "interesting" boolean from the event handler function on GFileMonitorSource. An event was "interesting" if it will result in a signal actually being dispatched to the user. It is "uninteresting" if it only hit an already-dirty rate limiter. commit 3d2d4b8efe44af1da7206b10e4565e59dbdb5121 Author: Ryan Lortie <desrt> Date: Thu Jan 15 11:08:35 2015 -0500 inotify: send CHANGES_DONE when new files 'appear' We generally assume that an IN_CREATE event is the start of a series of events in which another process is doing this: fd = creat (...) -> IN_CREATE write (fd, ..) -> IN_MODIFY write (fd, ..) -> IN_MODIFY close (fd) -> IN_CLOSE_WRITE and as such, we use the CHANGES_DONE_HINT event after CREATED in order to show when this sequence of events has completed (ie: when we receive IN_CLOSE_WRITE when the user closes the file). Renaming a file into place is handled by IN_MOVED_FROM so we don't have to worry about that. There are many other cases, however, where a new file 'appears' in a directory in its completed form already, and the kernel reports IN_CREATE. Examples include mkdir, mknod, and the creation of hardlinks. In these cases, there is no corresponding IN_CLOSE_WRITE event and the CHANGES_DONE_HINT will have to be emitted by an arbitrary timeout. Try to detect some of these cases and report CHANGES_DONE_HINT immediately. This is not perfect. There are some cases that will not be reliably detected. An example is if the user makes a hardlink and then immediately deletes the original (before we can stat the new file). Another example is if the user creates a file with O_TMPFILE. In both of these cases, CHANGES_DONE_HINT will still eventually be delivered via the timeout. commit 2737ab3201163631be152801a859b3874a667f10 Author: Ryan Lortie <desrt> Date: Mon Jan 12 14:59:35 2015 -0500 substantially rework file monitors Remove all event merging and dispatch logic from GFileMonitor. The only implementation of GFileMonitor outside of glib is in gvfs and it already does these things properly. Get rid of GLocalDirectoryMonitor. We will use a single class, GLocalFileMonitor, for both directory and file monitoring. This will prevent every single backend from having to create two objects separately (eg: ginotifydirectorymonitor.c and ginotifyfilemonitor.c). Introduce GFileMonitorSource as a thread-safe cross-context dispatch mechanism. Put it in GLocalFileMonitor. All backends will be expected to dispatch via the source and not touch the GFileMonitor object at all from the worker thread. Remove all construct properties from GLocalFileMonitor and remove the "context" construct property from GFileMonitor. All backends must now get the information about what file to monitor from the ->start() call which is mandatory to implement. Remove the implementation of rate limiting in GFileMonitor and add an implementation in GLocalFileMonitor. gvfs never did anything with this anyway, but if it wanted to, it would have to implement it for itself. This was done in order to get the rate_limit field into the GFileMonitorSource so that it could be safely accessed from the worker thread. Expose g_local_file_is_remote() internally for NFS detection. With the "is_remote" functionality exposed, we can now move all functions for creating local file monitors to a proper location in glocalfilemonitor.c Port the inotify backend to adjust to the changes above. None of the other backends are ported yet. Those will come in future commits. commit 779c809a3d07fca6c1da4f87d4ce6cf7f2d95608 Author: Ryan Lortie <desrt> Date: Thu Mar 5 21:28:29 2015 -0500 inotify: rewrite inotify-kernel Remove the hardwired 1 second event queue logic from inotify-kernel and replace it with something vastly less complicated. Events are now reported as soon as is possible instead of after a delay. We still must delay IN_MOVED_FROM events in order to look for the matching IN_MOVED_TO events, and since we want to report events in order this means that events behind those events can also be delayed. We limit ourselves, however: - no more than 100 events can be delayed at a time - no event can be delayed by more than 10ms https://bugzilla.gnome.org/show_bug.cgi?id=627285 Proposed as a Blocker for 23-beta by Fedora user stefw using the blocker tracking app because: This bug breaks a core feature of the GLib library: file monitoring. It affects cockpit, gnome-software, gnome-online-accounts, among others. For those following along, the Cockpit test suite has caught this bug in multiple places. Initially it was just Rawhide, but since the F23 branch, this obviously affects both. https://github.com/cockpit-project/cockpit/issues/2540 https://github.com/cockpit-project/cockpit/pull/2546 I'll try to find Ryan and get him to look into it This bug has prevented the Koji build of Cockpit 0.70 on F23 and Rawhide: http://koji.fedoraproject.org/koji/taskinfo?taskID=10681877 http://koji.fedoraproject.org/koji/taskinfo?taskID=10681871 This bug has prevented inclusion of the 0.71 Cockpit release in F23 and Rawhide. Still waiting for desrt to react, sorry. Maybe you can poke him too (In reply to Matthias Clasen from comment #10) > I'll try to find Ryan and get him to look into it He says he is on it: https://bugzilla.gnome.org/show_bug.cgi?id=751358#c6 This violates the Alpha criterion "after system installation the Cockpit web management interface must be running and accessible on its default port (9090)". Since the package cannot be built and run against the current version of Glib in stable on Fedora 23, this criterion is not met. +1 Beta Blocker. Discussed at 2015-08-20 blocker review meeting: http://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-08-20/f23-blocker-review.2015-08-20-16.05.log.txt . Accepted as a blocker per criterion cited in #c15. fixed in 2.45.5 yelp-xsl-3.17.90-1.fc23 yelp-3.17.90-1.fc23 webkitgtk4-2.9.90-1.fc23 tracker-1.5.2-1.fc23 totem-3.16.2-4.fc23 swell-foop-3.17.90-1.fc23 sound-juicer-3.17.90-1.fc23 simple-scan-3.17.90-1.fc23 rygel-0.27.4-1.fc23 pygobject3-3.17.90-1.fc23 pyatspi-2.17.90-1.fc23 polari-3.17.90-1.fc23 pango-1.37.3-1.fc23 orca-3.17.90-1.fc23 mutter-3.17.90-1.fc23 lightsoff-3.17.90-1.fc23 libvirt-glib-0.2.2-1.fc23 libsoup-2.51.90-1.fc23 libgtop2-2.31.90-1.fc23 latexila-3.17.0-1.fc23 iagno-3.17.90-1.fc23 gvfs-1.25.90-1.fc23 gucharmap-3.17.90-1.fc23 gtranslator-2.91.7-5.fc23 gtksourceview3-3.17.5-1.fc23 gtk3-3.17.7-1.fc23 gnome-user-share-3.17.90-1.fc23 gnome-tweak-tool-3.17.90-1.fc23 gnome-taquin-3.17.90-1.fc23 gnome-system-monitor-3.17.90-1.fc23 gnome-sudoku-3.17.90-1.fc23 gnome-sound-recorder-3.17.90-1.fc23 gnome-software-3.17.90-1.fc23 gnome-shell-extensions-3.17.90-1.fc23 gnome-shell-extension-background-logo-3.17.90-1.fc23 gnome-shell-3.17.90-1.fc23 gnome-settings-daemon-3.17.90-1.fc23 gnome-robots-3.17.90-1.fc23 gnome-power-manager-3.17.90-1.fc23 gnome-packagekit-3.17.90-1.fc23 gnome-nibbles-3.17.90-1.fc23 gnome-music-3.17.90-1.fc23 gnome-multi-writer-3.17.90-1.fc23 gnome-mines-3.17.90-1.fc23 gnome-maps-3.17.90-1.fc23 gnome-logs-3.17.90-1.fc23 gnome-klotski-3.17.90-1.fc23 gnome-initial-setup-3.17.90-1.fc23 gnome-dictionary-3.17.90-2.fc23 gnome-desktop3-3.17.90-1.fc23 gnome-contacts-3.16.2-7.fc23 gnome-color-manager-3.17.90-1.fc23 gnome-chess-3.17.90-1.fc23 gnome-characters-3.17.90-1.fc23 gnome-calendar-3.17.90-1.fc23 gnome-boxes-3.17.90-1.fc23 gnome-bluetooth-3.17.90-2.fc23 gnome-2048-3.17.90-1.fc23 glib2-2.45.6-1.fc23 geocode-glib-3.17.90-1.fc23 gedit-3.17.90-1.fc23 gdk-pixbuf2-2.31.6-1.fc23 fwupd-0.1.5-2.fc23 evolution-rspam-0.6.0-12.fc23 evolution-mapi-3.17.90-1.fc23 evolution-ews-3.17.90-1.fc23 evolution-data-server-3.17.90-1.fc23 evolution-3.17.90-1.fc23 eog-plugins-3.16.1-1.fc23 eog-3.17.90-1.fc23 empathy-3.12.10-4.fc23 dconf-editor-3.17.90-1.fc23 control-center-3.17.90-1.fc23 colord-1.2.12-1.fc23 cheese-3.17.90-1.fc23 bijiben-3.17.90-1.fc23 atomix-3.17.90-1.fc23 atk-2.17.90-1.fc23 at-spi2-core-2.17.90-1.fc23 at-spi2-atk-2.17.90-1.fc23 appstream-data-23-4.fc23 aisleriot-3.17.90-1.fc23 PackageKit-1.0.8-1.fc23 has been submitted as an update to Fedora 23. https://bugzilla.redhat.com/show_bug.cgi?id=1225957 (In reply to Matthias Clasen from comment #17) > fixed in 2.45.5 Great! I'll report back after running our tests. PackageKit-1.0.8-2.fc23, aisleriot-3.17.90-1.fc23, appstream-data-23-4.fc23, at-spi2-atk-2.17.90-1.fc23, at-spi2-core-2.17.90-1.fc23, atk-2.17.90-1.fc23, atomix-3.17.90-1.fc23, bijiben-3.17.90-1.fc23, cheese-3.17.90-1.fc23, clutter-1.23.6-1.fc23, cogl-1.21.2-2.fc23, colord-1.2.12-1.fc23, control-center-3.17.90-1.fc23, dconf-editor-3.17.90-1.fc23, empathy-3.12.10-4.fc23, eog-3.17.90-1.fc23, eog-plugins-3.16.1-1.fc23, evolution-3.17.90-1.fc23, evolution-data-server-3.17.90-1.fc23, evolution-ews-3.17.90-1.fc23, evolution-mapi-3.17.90-1.fc23, evolution-rspam-0.6.0-12.fc23, fwupd-0.1.5-3.fc23, gdk-pixbuf2-2.31.6-1.fc23, gedit-3.17.90-1.fc23, geocode-glib-3.17.90-1.fc23, glib2-2.45.6-1.fc23, gnome-2048-3.17.90-1.fc23, gnome-bluetooth-3.17.90-2.fc23, gnome-boxes-3.17.90-1.fc23, gnome-calendar-3.17.90-1.fc23, gnome-characters-3.17.90-1.fc23, gnome-chess-3.17.90-1.fc23, gnome-color-manager-3.17.90-1.fc23, gnome-contacts-3.16.2-7.fc23, gnome-desktop3-3.17.90-1.fc23, gnome-dictionary-3.17.90-2.fc23, gnome-documents-3.17.90-1.fc23, gnome-initial-setup-3.17.90-1.fc23, gnome-klotski-3.17.90-1.fc23, gnome-logs-3.17.90-1.fc23, gnome-maps-3.17.90-1.fc23, gnome-mines-3.17.90-1.fc23, gnome-multi-writer-3.17.90-1.fc23, gnome-music-3.17.90-1.fc23, gnome-nibbles-3.17.90-1.fc23, gnome-packagekit-3.17.90-1.fc23, gnome-power-manager-3.17.90-1.fc23, gnome-robots-3.17.90-1.fc23, gnome-settings-daemon-3.17.90-1.fc23, gnome-shell-3.17.90-1.fc23, gnome-shell-extension-background-logo-3.17.90-1.fc23, gnome-shell-extensions-3.17.90-1.fc23, gnome-software-3.17.90-1.fc23, gnome-sound-recorder-3.17.90-1.fc23, gnome-sudoku-3.17.90-1.fc23, gnome-system-monitor-3.17.90-1.fc23, gnome-taquin-3.17.90-1.fc23, gnome-tweak-tool-3.17.90-1.fc23, gnome-user-share-3.17.90-1.fc23, gobject-introspection-1.45.4-1.fc23, gtk3-3.17.7-1.fc23, gtksourceview3-3.17.5-1.fc23, gtranslator-2.91.7-5.fc23, gucharmap-3.17.90-1.fc23, gvfs-1.25.90-1.fc23, iagno-3.17.90-1.fc23, latexila-3.17.0-1.fc23, libgtop2-2.31.90-1.fc23, libsoup-2.51.90-1.fc23, libvirt-glib-0.2.2-1.fc23, lightsoff-3.17.90-1.fc23, mutter-3.17.90-1.fc23, nautilus-3.17.90-1.fc23, orca-3.17.90-1.fc23, pango-1.37.3-1.fc23, polari-3.17.90-1.fc23, pyatspi-2.17.90-1.fc23, pygobject3-3.17.90-1.fc23, rygel-0.27.4-1.fc23, simple-scan-3.17.90-1.fc23, sound-juicer-3.17.90-1.fc23, swell-foop-3.17.90-1.fc23, totem-3.16.2-4.fc23, tracker-1.5.2-1.fc23, webkitgtk4-2.9.90-1.fc23, yelp-3.17.90-1.fc23, yelp-xsl-3.17.90-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.\nIf you want to test the update, you can install it with \n su -c 'yum --enablerepo=updates-testing update gnome-mines cogl clutter gnome-documents fwupd nautilus PackageKit aisleriot appstream-data at-spi2-atk at-spi2-core atk atomix bijiben cheese colord control-center dconf-editor empathy eog eog-plugins evolution evolution-data-server evolution-ews evolution-mapi evolution-rspam gdk-pixbuf2 gedit geocode-glib glib2 gnome-2048 gnome-bluetooth gnome-boxes gnome-calendar gnome-characters gnome-chess gnome-color-manager gnome-contacts gnome-desktop3 gnome-dictionary gnome-initial-setup gnome-klotski gnome-logs gnome-maps gobject-introspection gnome-multi-writer gnome-music gnome-nibbles gnome-packagekit gnome-power-manager gnome-robots gnome-settings-daemon gnome-shell gnome-shell-extension-background-logo gnome-shell-extensions gnome-software gnome-sound-recorder gnome-sudoku gnome-system-monitor gnome-taquin gnome-tweak-tool gnome-user-share gtk3 gtksourceview3 gtranslator gucharmap gvfs iagno latexila libgtop2 libsoup libvirt-glib lightsoff mutter orca pango polari pyatspi pygobject3 rygel simple-scan sound-juicer swell-foop totem tracker webkitgtk4 yelp yelp-xsl'. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-13858 PackageKit-1.0.8-2.fc23, aisleriot-3.17.90-1.fc23, appstream-data-23-4.fc23, at-spi2-atk-2.17.90-1.fc23, at-spi2-core-2.17.90-1.fc23, atk-2.17.90-1.fc23, atomix-3.17.90-1.fc23, bijiben-3.17.90-1.fc23, cheese-3.17.90-1.fc23, clutter-1.23.6-1.fc23, cogl-1.21.2-2.fc23, colord-1.2.12-1.fc23, control-center-3.17.90-1.fc23, dconf-editor-3.17.90-1.fc23, empathy-3.12.10-4.fc23, eog-3.17.90-1.fc23, eog-plugins-3.16.1-1.fc23, evolution-3.17.90-1.fc23, evolution-data-server-3.17.90-1.fc23, evolution-ews-3.17.90-1.fc23, evolution-mapi-3.17.90-1.fc23, evolution-rspam-0.6.0-12.fc23, fwupd-0.1.5-3.fc23, gdk-pixbuf2-2.31.6-1.fc23, gedit-3.17.90-1.fc23, geocode-glib-3.17.90-1.fc23, glib2-2.45.6-1.fc23, gnome-2048-3.17.90-1.fc23, gnome-bluetooth-3.17.90-2.fc23, gnome-boxes-3.17.90-1.fc23, gnome-calendar-3.17.90-1.fc23, gnome-characters-3.17.90-1.fc23, gnome-chess-3.17.90-1.fc23, gnome-color-manager-3.17.90-1.fc23, gnome-contacts-3.16.2-7.fc23, gnome-desktop3-3.17.90-1.fc23, gnome-dictionary-3.17.90-2.fc23, gnome-documents-3.17.90-1.fc23, gnome-initial-setup-3.17.90-1.fc23, gnome-klotski-3.17.90-1.fc23, gnome-logs-3.17.90-1.fc23, gnome-maps-3.17.90-1.fc23, gnome-mines-3.17.90-1.fc23, gnome-multi-writer-3.17.90-1.fc23, gnome-music-3.17.90-1.fc23, gnome-nibbles-3.17.90-1.fc23, gnome-packagekit-3.17.90-1.fc23, gnome-power-manager-3.17.90-1.fc23, gnome-robots-3.17.90-1.fc23, gnome-settings-daemon-3.17.90-1.fc23, gnome-shell-3.17.90-1.fc23, gnome-shell-extension-background-logo-3.17.90-1.fc23, gnome-shell-extensions-3.17.90-1.fc23, gnome-software-3.17.90-1.fc23, gnome-sound-recorder-3.17.90-1.fc23, gnome-sudoku-3.17.90-1.fc23, gnome-system-monitor-3.17.90-1.fc23, gnome-taquin-3.17.90-1.fc23, gnome-tweak-tool-3.17.90-1.fc23, gnome-user-share-3.17.90-1.fc23, gobject-introspection-1.45.4-1.fc23, gtk3-3.17.7-1.fc23, gtksourceview3-3.17.5-1.fc23, gtranslator-2.91.7-5.fc23, gucharmap-3.17.90-1.fc23, gvfs-1.25.90-1.fc23, iagno-3.17.90-1.fc23, latexila-3.17.0-1.fc23, libgtop2-2.31.90-1.fc23, libsoup-2.51.90-1.fc23, libvirt-glib-0.2.2-1.fc23, lightsoff-3.17.90-1.fc23, mutter-3.17.90-1.fc23, nautilus-3.17.90-1.fc23, orca-3.17.90-1.fc23, pango-1.37.3-1.fc23, polari-3.17.90-1.fc23, pyatspi-2.17.90-1.fc23, pygobject3-3.17.90-1.fc23, rygel-0.27.4-1.fc23, simple-scan-3.17.90-1.fc23, sound-juicer-3.17.90-1.fc23, swell-foop-3.17.90-1.fc23, totem-3.16.2-4.fc23, tracker-1.5.2-1.fc23, webkitgtk4-2.9.90-1.fc23, yelp-3.17.90-1.fc23, yelp-xsl-3.17.90-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report. Now we see an additional "G_FILE_MONITOR_EVENT_DELETED" event for the other file ("bar" in the reproducer). I'll file another bug. (In reply to Marius Vollmer from comment #22) > Now we see an additional "G_FILE_MONITOR_EVENT_DELETED" event for the other > file ("bar" in the reproducer). I'll file another bug. Bug 1225957. (In reply to Marius Vollmer from comment #23) > (In reply to Marius Vollmer from comment #22) > > Now we see an additional "G_FILE_MONITOR_EVENT_DELETED" event for the other > > file ("bar" in the reproducer). I'll file another bug. > > Bug 1225957. Sorry, bug 1259594. (In reply to Marius Vollmer from comment #19) > Great! I'll report back after running our tests. The Cockpit integration tests pass now (or fail for other reasons). Thanks! |