Red Hat Bugzilla – Bug 336471
Flex generates "input in flex scanner failed" under variety of conditions.
Last modified: 2015-05-04 21:33:33 EDT
Description of problem: Flex-generated scanner produces same error under a variety of build options. Example response ---- $ ./fuzz -v ../tex/exampleUTF8.tex JPM: passed open_prelude() JPM: inside read_a_file .. passed yylloc JPM: entered newline filter zzlex. JPM: making a call to yylex in zzlex. --(end of buffer or a NUL) input in flex scanner failed $ ---- It does this with -m32, -m64, or neither gcc flags, and several plausible make and flex options as well as with or without the flex --nounistd switch. You can see my printf's but I can't insert them into flex generated code. The error appears to be an I/O problem, because I get the same error when trying to read non-existent files. That is, $fuzz nonexistent.tex produces the same "input in flex scanner failed" message rather than a "can't open" or "can't read" message that fuzz normally produces. I was able to build the fuzz tool on Mac OS X with no problems, so its not a fuzz issue, other than fuzz trying to be ANSI. Version-Release number of selected component (if applicable): ---- # rpm -qi flex Name : flex Relocations: (not relocatable) Version : 2.5.33 Vendor: Fedora Project Release : 9.fc7 Build Date: Mon 25 Jun 2007 10:34:08 AM EDT Install Date: Wed 17 Oct 2007 09:13:18 AM EDT Build Host: xenbuilder2.fedora.redhat.com Group : Development/Tools Source RPM: flex-2.5.33-9.fc7.src.rpm Size : 702392 License: BSD Signature : DSA/SHA1, Mon 25 Jun 2007 11:22:57 AM EDT, Key ID b44269d04f2a6fd2 Packager : Fedora Project URL : http://flex.sourceforge.net/ ---- # rpm -qi kernel Name : kernel Relocations: (not relocatable) Version : 2.6.22.5 Vendor: Fedora Project Release : 76.fc7 Build Date: Thu 30 Aug 2007 02:59:30 PM EDT Install Date: Fri 21 Sep 2007 08:19:12 AM EDT Build Host: xenbuilder4.fedora.phx.redhat.com Group : System Environment/Kernel Source RPM: kernel-2.6.22.5-76.fc7.src.rpm Size : 66412539 License: GPLv2 Signature : DSA/SHA1, Wed 12 Sep 2007 10:49:57 AM EDT, Key ID b44269d04f2a6fd2 Packager : Fedora Project How reproducible: Download the tarball for the fuzz tool and try to build it. Fuzz is available from http://spivey.oriel.ox.ac.uk/mike/fuzz/ and takes about 2 secs to build. Steps to Reproduce: 1. Initial attempts to build causes a unistd.h mismatch over the definition of read. 2. This problem can be removed by several approaches including commenting out, using the --nounistd switch for flex, and copying in the function prototypes from unistd.h. All of these compile but all produce the same error. 3. Actual results: Expected results: Additional info: I tried reverting to flex 5.4 but it still causes the same error. I suspect a problem with flex's approach to size_t, but that's beyond my knowledge of either flex or Fedora 7. Building fuzz does not reproduce this error on Mac OS X or Debian Sarge.
I tried to enter this as an FC7 bug, but Bugzilla would not give me fc7 as an option.
Created attachment 230201 [details] Fix for fuzz problem. The problem is in fuzz source code. They shouldn't declare 'read' function themselves but instead rely on libc providing it. The attached little patch (should be applied to fuzz, not flex!) fixes the build for me on x86_64, in both 32bit and 64bit mode. Build and testsuite then pass.
Hmm, as I'm re-reading your problem, it looks like you didn't generally complain that you can build it, but that fuzz can't read your tex file. I'll give it another look.
Thanks for looking at this. The patch pretty much does what I did at first, except I couldn't get a functional fuzz and still don't. If just removing the spurious read works for you, then my FC7 must be out of date in a way that I don't understand. I'll try even more updating of my host than I already have, and see if that makes the problem go away.
Well, without the patch I don't even build it. With the patch, I get few harmless warnings and a build which passes the testsuite. I've tried F7 as well as rawhide on x64, and F7 on x86. tex/example.tex gives me seemingly realistic output: "tex/example.tex", line 46: Type mismatch in left argument of infix relation > Predicate: president \subseteq members > Arg type: PERSON > Expected: P ? "tex/example.tex", line 54: Right argument of operator \cup has wrong type > Expression: members \cup new? > Arg type: PERSON > Expected: P PERSON So I can't seem to be able to reproduce this. Perhaps try to upload the exampleUTF8.tex file that exposes the failure, it doesn't come with the distribution of fuzz that I got from http://spivey.oriel.ox.ac.uk/mike/fuzz/.
I worked on this some more and have been able to get fUZZ to work properly. As far as I can tell, some part of my FC 7 was out-of-date, but I could not figure out which library or tool was the problem. I run KDE on the box in question, but I don't think KDE caused the problem. Since the problem can't be reproduced, it clearly is neither a flex nor an FC 7 bug.
The file exampleUTF8.tex was just a re-encoding of example.tex. I was trying various file encodings to see if that made any difference. I now have fUZZ working with both UTF8 and UTF16.
Ok, in that case I'll close the bug.