Bug 1232422

Summary: emacs "deadlocked" after using mercurial with huge amounts of ignored files in the repository
Product: [Fedora] Fedora Reporter: Garrett Mitchener <garrett.mitchener>
Component: emacsAssignee: Jan Synacek <jsynacek>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: garrett.mitchener, jchaloup, jonathan.underwood, phracek
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/abc2e24f8cf90ecf233772c99dcc9d7808f726ad
Whiteboard: abrt_hash:5c3826dc0986d72c33c34f916f2a1afab70eb9cb
Fixed In Version: emacs-24.5-7.fc23 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-04 23:22:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: cgroup
none
File: core_backtrace
none
File: dso_list
none
File: environ
none
File: limits
none
File: maps
none
File: mountinfo
none
File: namespaces
none
File: open_fds
none
File: proc_pid_status none

Description Garrett Mitchener 2015-06-16 17:42:45 UTC
Description of problem:
I'm trying to track down a bug somewhere in emacs or maybe mercurial. I can reproduce the problem as follows.

1. open emacs in a directory with a .hg subdirectory
2. C-x v v  Enter,  to open up a VC dir buffer viewing that directory
3. go to a file that's been modified and hit v, enter a log message, then C-c C-c to commit the change
4. hit g to refresh the buffer -- then something goes wrong and the status line says [waiting....]

From that point on, emacs is unresponsive. I used pkill emacs to stop it and trigger ABRT to make this bug report

Version-Release number of selected component:
emacs-24.5-2.fc22

Additional info:
reporter:       libreport-2.5.1
backtrace_rating: 4
cmdline:        emacs APSTest.hs
crash_function: AREF
executable:     /usr/bin/emacs-24.5
global_pid:     32400
kernel:         4.0.4-303.fc22.x86_64
runlevel:       N 5
type:           CCpp
uid:            1000
var_log_messages: [System Logs]:\n-- Logs begin at Wed 2015-06-03 16:18:33 EDT, end at Tue 2015-06-16 13:30:49 EDT. --

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 AREF at ../../src/lisp.h:1356
 #1 HASH_KEY at ../../src/lisp.h:1814
 #2 hash_lookup at ../../src/fns.c:3813
 #3 Fcoding_system_p at ../../src/coding.c:8504
 #4 Fcheck_coding_system at ../../src/coding.c:8563
 #5 Ffuncall at ../../src/eval.c:2811
 #6 exec_byte_code at ../../src/bytecode.c:916
 #7 funcall_lambda at ../../src/eval.c:3044
 #8 Ffuncall at ../../src/eval.c:2872
 #9 exec_byte_code at ../../src/bytecode.c:916

Comment 1 Garrett Mitchener 2015-06-16 17:42:47 UTC
Created attachment 1039601 [details]
File: backtrace

Comment 2 Garrett Mitchener 2015-06-16 17:42:48 UTC
Created attachment 1039602 [details]
File: cgroup

Comment 3 Garrett Mitchener 2015-06-16 17:42:49 UTC
Created attachment 1039603 [details]
File: core_backtrace

Comment 4 Garrett Mitchener 2015-06-16 17:42:50 UTC
Created attachment 1039604 [details]
File: dso_list

Comment 5 Garrett Mitchener 2015-06-16 17:42:51 UTC
Created attachment 1039605 [details]
File: environ

Comment 6 Garrett Mitchener 2015-06-16 17:42:52 UTC
Created attachment 1039606 [details]
File: limits

Comment 7 Garrett Mitchener 2015-06-16 17:42:53 UTC
Created attachment 1039607 [details]
File: maps

Comment 8 Garrett Mitchener 2015-06-16 17:42:53 UTC
Created attachment 1039608 [details]
File: mountinfo

Comment 9 Garrett Mitchener 2015-06-16 17:42:54 UTC
Created attachment 1039609 [details]
File: namespaces

Comment 10 Garrett Mitchener 2015-06-16 17:42:55 UTC
Created attachment 1039610 [details]
File: open_fds

Comment 11 Garrett Mitchener 2015-06-16 17:42:56 UTC
Created attachment 1039611 [details]
File: proc_pid_status

Comment 12 Garrett Mitchener 2015-06-16 17:44:38 UTC
Oops, make that C-x v d Enter to view the directory.

Comment 13 Fedora Admin XMLRPC Client 2015-12-15 11:27:20 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 14 Jan Synacek 2016-01-05 08:48:49 UTC
I can't reproduce this. Can you still make the reproducer work when running "emacs -Q" ?

Comment 15 Garrett Mitchener 2016-01-25 19:58:18 UTC
Sorry, I think I made a mistake on item 2:

1. open emacs in a directory with a .hg subdirectory
2. C-x v d  Enter,  to open up a VC dir buffer viewing that directory
3. go to a file that's been modified and hit v, enter a log message, then C-c C-c to commit the change
4. hit g to refresh the buffer -- then something goes wrong and the status line says [waiting...]

I just tried it again on an up-to-date Fedora 23 machine. Emacs is more responsive, but the [waiting...] never disappears and the update process in the *vc-dir* buffer takes forever to finish.

The command running in that buffer is

hg status -mardui -C files...

I think the source of the problem is hg status -i.  Some of my directories include a *lot* of ignored files, and previously emacs would hang waiting for hg to print information about every one of them.

So I think that as of emacs-24.5-6.fc23.x86_64, the problem has mostly been fixed. What's left is that vc-dir.el and vc-hg.el do a lot of work that doesn't seem to be necessary in directories like that.

Comment 16 Jan Synacek 2016-01-27 14:05:23 UTC
(In reply to Garrett Mitchener from comment #15)
> hg status -mardui -C files...
> 
> I think the source of the problem is hg status -i.  Some of my directories
> include a *lot* of ignored files, and previously emacs would hang waiting
> for hg to print information about every one of them.

This is what is indeed causing a delay. How many ignored files do you have? I have generated 10000 ignored files and the delay is still less than 1 second. But I agree that this should be fixed, there should probably be an upper limit on how many files should be displayed.

> So I think that as of emacs-24.5-6.fc23.x86_64, the problem has mostly been
> fixed. What's left is that vc-dir.el and vc-hg.el do a lot of work that
> doesn't seem to be necessary in directories like that.

On Fedora 23, you can try https://copr.fedorainfracloud.org/coprs/jsynacek/emacs-25/. The code looks pretty similar, though, at least in vc-hg.el.

Comment 17 Garrett Mitchener 2016-01-27 16:30:09 UTC
[garrett@grograman]$ time hg status -i | wc
6890134 13782556 464679469

real	9m0.809s
user	6m7.277s
sys	0m59.760s

So yes, almost 7 million.  I have a *lot* of ignored files...

This is a simulation project, and those are data files that shouldn't be tracked by hg.  But I do need for several scripts to be under version control.

Comment 18 Jan Synacek 2016-01-28 08:27:30 UTC
Ok, so this looks like something that should be correctly handled in hg. It takes a lot of time to process the files.

You can apply the following workaround:
1) sudo emacs /usr/share/emacs/24.5/lisp/vc/vc-hg.el.gz
2) Apply the following patch to the file (simply remove the 'i' from hg arguments):
@@ -628,7 +628,7 @@
    (vc-hg-after-dir-status update-function)))
 
 (defun vc-hg-dir-status-files (dir files _default-state update-function)
-  (apply 'vc-hg-command (current-buffer) 'async dir "status" "-mardui" "-C" files)
+  (apply 'vc-hg-command (current-buffer) 'async dir "status" "-mardu" "-C" files)
   (vc-run-delayed
    (vc-hg-after-dir-status update-function)))
3) Save and restart emacs (or reevaluate vc-hg-dir-status-files).
 
I can't think of anything better right now. Hg takes way too much time displaying those ignored files and emacs just bluntly waits and tries to display all of them.

Comment 19 Jan Synacek 2016-01-28 09:16:44 UTC
Reported upstream:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22481

Comment 21 Fedora Update System 2016-02-03 08:55:03 UTC
emacs-24.5-7.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-37ae1c1888

Comment 22 Fedora Update System 2016-02-03 23:00:34 UTC
emacs-24.5-7.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-37ae1c1888

Comment 23 Fedora Update System 2016-02-04 23:22:17 UTC
emacs-24.5-7.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.