Bug 613657

Summary: [abrt] crash in pylint-0.20.0-1.fc13: tokenize.py:352:generate_tokens:TokenError: ('EOF in multi-line statement', (3093, 0))
Product: [Fedora] Fedora Reporter: Mads Kiilerich <mads>
Component: pylintAssignee: Brian Lane <bcl>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 13CC: bcl, mrunge
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard: abrt_hash:b3e572e3
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-07-13 16:27:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace none

Description Mads Kiilerich 2010-07-12 14:13:51 UTC
abrt 1.1.1 detected a crash.

architecture: i686
cmdline: /usr/bin/python -u /usr/bin/pylint --include-ids=y --disable-msg=W0511,W0612,W0603,W0142 /home/mk/src/hg/hgext/mq.py
component: pylint
executable: /usr/bin/pylint
kernel: 2.6.33.6-147.fc13.i686.PAE
package: pylint-0.20.0-1.fc13
reason: tokenize.py:352:generate_tokens:TokenError: ('EOF in multi-line statement', (3093, 0))
release: Fedora release 13 (Goddard)

backtrace
-----
tokenize.py:352:generate_tokens:TokenError: ('EOF in multi-line statement', (3093, 0))

Traceback (most recent call last):
  File "/usr/bin/pylint", line 4, in <module>
    lint.Run(sys.argv[1:])
  File "/usr/lib/python2.6/site-packages/pylint/lint.py", line 885, in __init__
    linter.check(args)
  File "/usr/lib/python2.6/site-packages/pylint/lint.py", line 496, in check
    self.check_astng_module(astng, checkers)
  File "/usr/lib/python2.6/site-packages/pylint/lint.py", line 570, in check_astng_module
    checker.process_module(stream)
  File "/usr/lib/python2.6/site-packages/pylint/checkers/__init__.py", line 125, in process_module
    self.process_tokens(tokenize.generate_tokens(stream.readline))
  File "/usr/lib/python2.6/site-packages/pylint/checkers/format.py", line 206, in process_tokens
    for (tok_type, token, start, _, line) in tokens:
  File "/usr/lib/python2.6/tokenize.py", line 352, in generate_tokens
    raise TokenError, ("EOF in multi-line statement", (lnum, 0))
TokenError: ('EOF in multi-line statement', (3093, 0))

Local variables in innermost frame:
lnum: 3093
pos: 0
namechars: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_'
readline: <built-in method readline of file object at 0x91dad40>
strstart: (2675, 4)
epos: (3092, 49)
end: 49
initial: '\n'
needcont: 0
contstr: ''
start: 48
max: 0
endmatch: <_sre.SRE_Match object at 0x91b82c0>
spos: (3092, 48)
indents: [0, 4, 8, 12]
contline: None
line: ''
continued: 0
comment_token: '# Otherwise, they should form a linear path to a head.'
pseudomatch: <_sre.SRE_Match object at 0x83fc7b8>
column: 12
nl_pos: 66
endprog: <_sre.SRE_Pattern object at 0x8121658>
token: '\n'
numchars: '0123456789'
parenlev: 1

Comment 1 Mads Kiilerich 2010-07-12 14:13:54 UTC
Created attachment 431187 [details]
File: backtrace

Comment 2 Brian Lane 2010-07-12 21:57:07 UTC
Could you attach mq.py as well?

Comment 3 Mads Kiilerich 2010-07-13 01:22:23 UTC
It seems like the problem happened because I was updating (truncating) files (tracked by mercurial) while pylint was checking them, and pylint thus got so confused that it crashed.

I will try to fix Mercurial so it don't modify the files, but the issue here is that pylint should handle such errors with a (fatal) error message but without crashing.

Comment 4 Mads Kiilerich 2010-07-13 12:27:48 UTC
Reproducibel with

f=open('x.py', 'w')
f.write('+1\\\n'*100)
import os
os.system('pylint x.py &')
for i in range(f.tell(),1,-4):
    print i
    f.seek(i)
    f.truncate()

Comment 5 Brian Lane 2010-07-13 16:27:12 UTC
That's not something that I would expect to work.

Comment 6 Mads Kiilerich 2010-07-13 17:26:33 UTC
(In reply to comment #5)
> That's not something that I would expect to work.    

That's fair enough. But it happens in real life, and it gives a crash that abrt will report to you (or CC here).

I would expect it to fail in a civilized way.