Bug 336471

Summary: Flex generates "input in flex scanner failed" under variety of conditions.
Product: [Fedora] Fedora Reporter: John McDermott <john.mcdermott>
Component: flexAssignee: Petr Machata <pmachata>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: low    
Version: 7CC: mnewsome
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2007-10-25 17:04:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Fix for fuzz problem. none

Description John McDermott 2007-10-17 17:55:19 UTC 
Description of problem: Flex-generated scanner produces same error under a
variety of build options. Example response
----
$ ./fuzz -v ../tex/exampleUTF8.tex
JPM: passed open_prelude()
JPM: inside read_a_file .. passed yylloc
JPM: entered newline filter zzlex.
JPM: making a call to yylex in zzlex.
--(end of buffer or a NUL)
input in flex scanner failed
$
----

It does this with -m32, -m64, or neither gcc flags, and several plausible make
and flex options as well as with or without the flex --nounistd switch. You can
see my printf's but I can't insert them into flex generated code. The error
appears to be an I/O problem, because I get the same error when trying to read
non-existent files. That is, $fuzz nonexistent.tex produces the same "input in
flex scanner failed" message rather than a "can't open" or "can't read" message
that fuzz normally produces. I was able to build the fuzz tool on Mac OS X with
no problems, so its not a fuzz issue, other than fuzz trying to be ANSI.

Version-Release number of selected component (if applicable):
----
# rpm -qi flex
Name        : flex                         Relocations: (not relocatable)
Version     : 2.5.33                            Vendor: Fedora Project
Release     : 9.fc7                         Build Date: Mon 25 Jun 2007 10:34:08
AM EDT
Install Date: Wed 17 Oct 2007 09:13:18 AM EDT      Build Host:
xenbuilder2.fedora.redhat.com
Group       : Development/Tools             Source RPM: flex-2.5.33-9.fc7.src.rpm
Size        : 702392                           License: BSD
Signature   : DSA/SHA1, Mon 25 Jun 2007 11:22:57 AM EDT, Key ID b44269d04f2a6fd2
Packager    : Fedora Project
URL         : http://flex.sourceforge.net/
----
# rpm -qi kernel
Name        : kernel                       Relocations: (not relocatable)
Version     : 2.6.22.5                          Vendor: Fedora Project
Release     : 76.fc7                        Build Date: Thu 30 Aug 2007 02:59:30
PM EDT
Install Date: Fri 21 Sep 2007 08:19:12 AM EDT      Build Host:
xenbuilder4.fedora.phx.redhat.com
Group       : System Environment/Kernel     Source RPM:
kernel-2.6.22.5-76.fc7.src.rpm
Size        : 66412539                         License: GPLv2
Signature   : DSA/SHA1, Wed 12 Sep 2007 10:49:57 AM EDT, Key ID b44269d04f2a6fd2
Packager    : Fedora Project


How reproducible:

Download the tarball for the fuzz tool and try to build it. Fuzz is available
from http://spivey.oriel.ox.ac.uk/mike/fuzz/ and takes about 2 secs to build.

Steps to Reproduce:
1. Initial attempts to build causes a unistd.h mismatch over the definition of read.
2. This problem can be removed by several approaches including commenting out,
using the --nounistd switch for flex, and copying in the function prototypes
from unistd.h. All of these compile but all produce the same error.
3.
  
Actual results:


Expected results:


Additional info:

I tried reverting to flex 5.4 but it still causes the same error. I suspect a
problem with flex's approach to size_t, but that's beyond my knowledge of either
flex or Fedora 7.

Building fuzz does not reproduce this error on Mac OS X or Debian Sarge.
Comment 1 John McDermott 2007-10-17 17:56:42 UTC 
I tried to enter this as an FC7 bug, but Bugzilla would not give me fc7 as an
option.
Comment 2 Petr Machata 2007-10-17 18:41:02 UTC 
Created attachment 230201 [details]
Fix for fuzz problem.

The problem is in fuzz source code.  They shouldn't declare 'read' function
themselves but instead rely on libc providing it.  The attached little patch
(should be applied to fuzz, not flex!) fixes the build for me on x86_64, in
both 32bit and 64bit mode.  Build and testsuite then pass.
Comment 3 Petr Machata 2007-10-17 18:50:49 UTC 
Hmm, as I'm re-reading your problem, it looks like you didn't generally complain
that you can build it, but that fuzz can't read your tex file.  I'll give it
another look.
Comment 4 John McDermott 2007-10-17 19:02:36 UTC 
Thanks for looking at this. The patch pretty much does what I did at first,
except I couldn't get a functional fuzz and still don't. If just removing the
spurious read works for you, then my FC7 must be out of date in a way that I
don't understand. I'll try even more updating of my host than I already have,
and see if that makes the problem go away.
Comment 5 Petr Machata 2007-10-17 19:15:08 UTC 
Well, without the patch I don't even build it.  With the patch, I get few
harmless warnings and a build which passes the testsuite.  I've tried F7 as well
as rawhide on x64, and F7 on x86.  tex/example.tex gives me seemingly realistic
output:

"tex/example.tex", line 46: Type mismatch in left argument of infix
        relation
> Predicate: president \subseteq members
> Arg type:  PERSON
> Expected:  P ?

"tex/example.tex", line 54: Right argument of operator \cup has wrong
        type
> Expression: members \cup new?
> Arg type:   PERSON
> Expected:   P PERSON

So I can't seem to be able to reproduce this.  Perhaps try to upload the
exampleUTF8.tex file that exposes the failure, it doesn't come with the
distribution of fuzz that I got from http://spivey.oriel.ox.ac.uk/mike/fuzz/.
Comment 6 John McDermott 2007-10-22 13:53:09 UTC 
I worked on this some more and have been able to get fUZZ to work properly. As far as I can tell, some 
part of my FC 7 was out-of-date, but I could not figure out which library or tool was the problem. I run 
KDE on the box in question, but I don't think KDE caused the problem. Since the problem can't be 
reproduced, it clearly is neither a flex nor an FC 7 bug.
Comment 7 John McDermott 2007-10-22 13:56:02 UTC 
The file exampleUTF8.tex was just a re-encoding of example.tex. I was trying various file encodings to see 
if that made any difference. I now have fUZZ working with both UTF8 and UTF16.
Comment 8 Petr Machata 2007-10-25 17:04:18 UTC 
Ok, in that case I'll close the bug.