Bug 862079

Summary: [abrt] python-pep8-1.0.1-1.fc17: tokenize.py:357:generate_tokens:TokenError: ('EOF in multi-line statement', (7, 0))
Product: [Fedora] Fedora Reporter: MartinG <martin>
Component: python-pep8Assignee: Ian Weller <ian>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 17CC: ian, noelduffy
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:f171d841baabec7bf7ed381de0b6b006c5432248
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-30 03:16:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: core_backtrace none

Description MartinG 2012-10-01 19:44:32 UTC
Description of problem:
I'm using emacs with flymake for .py files. While typing, this package crashed.

This is the relevant code in my ~/.emacs

;;# yum install pyflakes python-pep8
;;~/.emacs.d/site-lisp:$ wget http://www.emacswiki.org/emacs/flymake-cursor.el
;;(add-to-list 'load-path "~/.emacs.d/site-lisp/")


(add-hook 'find-file-hook 'flymake-find-file-hook)
(when (load "flymake" t)
  (defun flymake-pyflakes-init ()
    (let* ((temp-file (flymake-init-create-temp-buffer-copy
		       'flymake-create-temp-inplace))
	   (local-file (file-relative-name
			temp-file
			(file-name-directory buffer-file-name))))
      (list "pycheckers"  (list local-file))))
     ;;(list "pylint"  (list local-file))))
  (add-to-list 'flymake-allowed-file-name-masks
	       '("\\.py\\'" flymake-pyflakes-init)))
(load-library "flymake-cursor")

The (invalid) code which caused the crash was:
#!/bin/python

if __name__ == "__main__":
    for i in range(1,100):
        print(i)


This is on a Fedora 17 x86_64 system.


Version-Release number of selected component:
python-pep8-1.0.1-1.fc17

Additional info:
libreport version: 2.0.14
abrt_version:   2.0.13
kernel:         3.5.4-1.fc17.x86_64

backtrace:
:tokenize.py:357:generate_tokens:TokenError: ('EOF in multi-line statement', (7, 0))
:
:Traceback (most recent call last):
:  File "/usr/bin/pep8", line 9, in <module>
:    load_entry_point('pep8==1.0.1', 'console_scripts', 'pep8')()
:  File "/usr/lib/python2.7/site-packages/pep8.py", line 1348, in _main
:    runner(path)
:  File "/usr/lib/python2.7/site-packages/pep8.py", line 1023, in input_file
:    errors = Checker(filename).check_all()
:  File "/usr/lib/python2.7/site-packages/pep8.py", line 949, in check_all
:    for token in tokenize.generate_tokens(self.readline_check_physical):
:  File "/usr/lib64/python2.7/tokenize.py", line 357, in generate_tokens
:    raise TokenError, ("EOF in multi-line statement", (lnum, 0))
:TokenError: ('EOF in multi-line statement', (7, 0))
:
:Local variables in innermost frame:
:lnum: 7
:pos: 0
:namechars: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_'
:readline: <bound method Checker.readline_check_physical of <pep8.Checker object at 0x1c83790>>
:epos: (6, 15)
:end: 15
:initial: '\n'
:needcont: 0
:contstr: ''
:start: 14
:max: 0
:spos: (6, 14)
:indents: [0, 4, 8]
:contline: None
:line: ''
:continued: 0
:comment_token: '#!/bin/python'
:pseudomatch: <_sre.SRE_Match object at 0x1d0a240>
:column: 8
:nl_pos: 13
:token: '\n'
:numchars: '0123456789'
:parenlev: 1

Comment 1 MartinG 2012-10-01 19:44:35 UTC
Created attachment 619975 [details]
File: core_backtrace

Comment 2 Ian Weller 2013-01-30 03:16:48 UTC
This crash report appears to be for running pep8 on obviously incorrect code (i.e. syntax errors), and a traceback is to be expected.

Reporter, you may reopen this ticket -- with more information on why this is a bug -- if you believe the above is incorrect.

Comment 3 Noel Duffy 2013-03-19 04:31:01 UTC
This is absolutely a bug. The whole point of pep8 is that you run it on your Python script and it reports errors, syntactical or otherwise, bugs, and style problems. Having it crash on an input script just because that script is syntactically wrong is the equivalent of GCC crashing if fed a syntactically wrong C program.