Bug 336471 - Flex generates "input in flex scanner failed" under variety of conditions.
Summary: Flex generates "input in flex scanner failed" under variety of conditions.
Alias: None
Product: Fedora
Classification: Fedora
Component: flex   
(Show other bugs)
Version: 7
Hardware: x86_64 Linux
Target Milestone: ---
Assignee: Petr Machata
QA Contact: Fedora Extras Quality Assurance
Keywords: Reopened
Depends On:
TreeView+ depends on / blocked
Reported: 2007-10-17 17:55 UTC by John McDermott
Modified: 2015-05-05 01:33 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-10-25 17:04:18 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Fix for fuzz problem. (411 bytes, patch)
2007-10-17 18:41 UTC, Petr Machata
no flags Details | Diff

Description John McDermott 2007-10-17 17:55:19 UTC
Description of problem: Flex-generated scanner produces same error under a
variety of build options. Example response
$ ./fuzz -v ../tex/exampleUTF8.tex
JPM: passed open_prelude()
JPM: inside read_a_file .. passed yylloc
JPM: entered newline filter zzlex.
JPM: making a call to yylex in zzlex.
--(end of buffer or a NUL)
input in flex scanner failed

It does this with -m32, -m64, or neither gcc flags, and several plausible make
and flex options as well as with or without the flex --nounistd switch. You can
see my printf's but I can't insert them into flex generated code. The error
appears to be an I/O problem, because I get the same error when trying to read
non-existent files. That is, $fuzz nonexistent.tex produces the same "input in
flex scanner failed" message rather than a "can't open" or "can't read" message
that fuzz normally produces. I was able to build the fuzz tool on Mac OS X with
no problems, so its not a fuzz issue, other than fuzz trying to be ANSI.

Version-Release number of selected component (if applicable):
# rpm -qi flex
Name        : flex                         Relocations: (not relocatable)
Version     : 2.5.33                            Vendor: Fedora Project
Release     : 9.fc7                         Build Date: Mon 25 Jun 2007 10:34:08
Install Date: Wed 17 Oct 2007 09:13:18 AM EDT      Build Host:
Group       : Development/Tools             Source RPM: flex-2.5.33-9.fc7.src.rpm
Size        : 702392                           License: BSD
Signature   : DSA/SHA1, Mon 25 Jun 2007 11:22:57 AM EDT, Key ID b44269d04f2a6fd2
Packager    : Fedora Project
URL         : http://flex.sourceforge.net/
# rpm -qi kernel
Name        : kernel                       Relocations: (not relocatable)
Version     :                          Vendor: Fedora Project
Release     : 76.fc7                        Build Date: Thu 30 Aug 2007 02:59:30
Install Date: Fri 21 Sep 2007 08:19:12 AM EDT      Build Host:
Group       : System Environment/Kernel     Source RPM:
Size        : 66412539                         License: GPLv2
Signature   : DSA/SHA1, Wed 12 Sep 2007 10:49:57 AM EDT, Key ID b44269d04f2a6fd2
Packager    : Fedora Project

How reproducible:

Download the tarball for the fuzz tool and try to build it. Fuzz is available
from http://spivey.oriel.ox.ac.uk/mike/fuzz/ and takes about 2 secs to build.

Steps to Reproduce:
1. Initial attempts to build causes a unistd.h mismatch over the definition of read.
2. This problem can be removed by several approaches including commenting out,
using the --nounistd switch for flex, and copying in the function prototypes
from unistd.h. All of these compile but all produce the same error.
Actual results:

Expected results:

Additional info:

I tried reverting to flex 5.4 but it still causes the same error. I suspect a
problem with flex's approach to size_t, but that's beyond my knowledge of either
flex or Fedora 7.

Building fuzz does not reproduce this error on Mac OS X or Debian Sarge.

Comment 1 John McDermott 2007-10-17 17:56:42 UTC
I tried to enter this as an FC7 bug, but Bugzilla would not give me fc7 as an

Comment 2 Petr Machata 2007-10-17 18:41:02 UTC
Created attachment 230201 [details]
Fix for fuzz problem.

The problem is in fuzz source code.  They shouldn't declare 'read' function
themselves but instead rely on libc providing it.  The attached little patch
(should be applied to fuzz, not flex!) fixes the build for me on x86_64, in
both 32bit and 64bit mode.  Build and testsuite then pass.

Comment 3 Petr Machata 2007-10-17 18:50:49 UTC
Hmm, as I'm re-reading your problem, it looks like you didn't generally complain
that you can build it, but that fuzz can't read your tex file.  I'll give it
another look.

Comment 4 John McDermott 2007-10-17 19:02:36 UTC
Thanks for looking at this. The patch pretty much does what I did at first,
except I couldn't get a functional fuzz and still don't. If just removing the
spurious read works for you, then my FC7 must be out of date in a way that I
don't understand. I'll try even more updating of my host than I already have,
and see if that makes the problem go away.

Comment 5 Petr Machata 2007-10-17 19:15:08 UTC
Well, without the patch I don't even build it.  With the patch, I get few
harmless warnings and a build which passes the testsuite.  I've tried F7 as well
as rawhide on x64, and F7 on x86.  tex/example.tex gives me seemingly realistic

"tex/example.tex", line 46: Type mismatch in left argument of infix
> Predicate: president \subseteq members
> Arg type:  PERSON
> Expected:  P ?

"tex/example.tex", line 54: Right argument of operator \cup has wrong
> Expression: members \cup new?
> Arg type:   PERSON
> Expected:   P PERSON

So I can't seem to be able to reproduce this.  Perhaps try to upload the
exampleUTF8.tex file that exposes the failure, it doesn't come with the
distribution of fuzz that I got from http://spivey.oriel.ox.ac.uk/mike/fuzz/.

Comment 6 John McDermott 2007-10-22 13:53:09 UTC
I worked on this some more and have been able to get fUZZ to work properly. As far as I can tell, some 
part of my FC 7 was out-of-date, but I could not figure out which library or tool was the problem. I run 
KDE on the box in question, but I don't think KDE caused the problem. Since the problem can't be 
reproduced, it clearly is neither a flex nor an FC 7 bug.

Comment 7 John McDermott 2007-10-22 13:56:02 UTC
The file exampleUTF8.tex was just a re-encoding of example.tex. I was trying various file encodings to see 
if that made any difference. I now have fUZZ working with both UTF8 and UTF16.

Comment 8 Petr Machata 2007-10-25 17:04:18 UTC
Ok, in that case I'll close the bug.

Note You need to log in before you can comment on or make changes to this bug.