Bug 1283275

+++ This bug was initially created as a clone of Bug #304121 +++

Description of problem:
When a the source directory for a file ends with a '/', debugedit can spit out a
"canonicalization unexpectedly shrank by one character" error and fail.  That's
not really enough for most people to figure out what's gone wrong or how to fix it.

Version-Release number of selected component (if applicable):
rpm-build-4.4.2.2-0.5.rc2

How reproducible:
Always

Steps to Reproduce:
1. mkdir src
2. cat > src/foo.c <<- EOF
   int
   main(int argc, char **argv)
   {
        return 0;
   }
EOF
3. cd src
4. gcc -g3 -o foo ../src//foo.c
5. /usr/lib/rpm/debugedit -b `pwd` -d / foo
  
Actual results:
/usr/lib/rpm/debugedit: canonicalization unexpectedly shrank by one character

Expected results:
Either not fail, or indicate that a directory name in the table ended with a
'/', which it probably shouldn't have contained.

Additional info:
I actually ran into this with openafs 1.4.4, but I'm including a smaller
reproducer which I think triggers the same bug.

--- Additional comment from Panu Matilainen on 2007-09-27 04:49:46 EDT ---

Hmm, it's not the trailing '/' that troubles debugedit per se but the fact that
the canonicalized path is exactly one character shorter than the original which
is a case that it can't (and can't be made to AFAICT) handle. Roland, care to
comment?

The error message could certainly be clearer on what sort of problem to look for
(basically single extra '/' somewhere in the path).

--- Additional comment from Roland McGrath on 2007-09-27 19:26:18 EDT ---

Indeed, the format makes it impossible to do rewriting in place unless the table
shrinks by at least 2 or not at all (it's beyond the scope of debugedit to
resize things).  For cases in the directory table you could do an ugly kludge by
repeating the leading / or adding a trailing one to preserve the size.  But
there is no trick like that to do in the file table, where any leading or
trailing slash would change the final file name so it doesn't work (not just is
wrongly ugly).  The !=-1 size restriction is for the whole table cumulatively,
so with extra slashes removed by canonicalization, it is only a problem when
there is only one such file name.  It could do a kludge of adding a trailing
slash to some dir table entry when this arises, which would work and only uglify
some of the names to /foo//bar.  Still, I suspect packagers will be OK with it
being an error they have to fix or work around, if they can tell what to do.

The error message you get now is certainly pretty arcane.  The other error of
the same ilk has a better message.  

Looking at canonicalize_path, I think it is true that a doubled / is the only
way it could shrink by exactly one.  But I could use a second set of eyes to
verify there really is no obscure innocent case that could do that.  If that's
true, then we can simply change the error message to identify this problem.

--- Additional comment from Jan Iven on 2007-11-14 05:53:44 EST ---

FYI, the instance that triggers this bug has apparently been "fixed" in the
Fedora-8-specific OpenAFS SRPM at
http://openafs.org/dl/openafs/1.4.5/fedora-8/SRPMS/openafs-1.4.5-fc8.2.src.rpm
Nonetheless, this could of course affect other packages as well..


--- Additional comment from Jason Tibbitts on 2007-12-06 18:43:46 EST ---

Just a note that a koji build for syslog-ng failed with the same error:
http://koji.fedoraproject.org/koji/taskinfo?taskID=265344

I'm not sure what would have changed to cause this package to stop building in
this manner.

--- Additional comment from Roland McGrath on 2007-12-06 20:21:12 EST ---

This line from the log suggests the problem:

/home/bazsi/zwa/git//syslog-ng/syslog-ng--mainline--2.0/src/cfg-lex.l:247:
warning: ignoring return value of 'fwrite', declared with attribute
warn_unused_result

The common reason for this error is // instead of / inside a file name.
This arises from using a trailing / on a directory name in a macro or makefile
variable usually.  

It may be a further problem that this absolute directory name appears in the
build.  Presumably it is in # line file name strings in cfg-lex.c, which is
generated from cfg-lex.l.  If this file is included in the upstream tarball like
this, then so it is and that is fine from the rpm-reproducible-builds
perspective.  Even so, upstream should be taught how to generate genericized
file names for their dist tarballs, so it winds up just # line "cfg-lex.l" or
suchlike.  If the rpm build is running lex/flex to generate cfg-lex.c itself,
then it's just our spec file that has to be fixed.

--- Additional comment from Douglas E. Warner on 2008-01-08 11:51:04 EST ---

I've created a patch for syslog-ng that fixes the double slash in the 
cfg-lex.c and cfg-grammar.c files but I still get the error with debuginfo and 
the warning referenced above as well.

I'm sorry I'm not much more help with this; I still don't fully understand 
what the error is here.

--- Additional comment from Bug Zapper on 2008-05-13 23:16:33 EDT ---

Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

--- Additional comment from Orion Poplawski on 2008-05-27 13:40:58 EDT ---

I'm seeing the same problem, but don't see any '//' occurrences.  See
http://koji.fedoraproject.org/koji/getfile?taskID=630760&name=build.log

If anyone has any suggestions as to how to fix, I would appreciate it.

--- Additional comment from Roland McGrath on 2008-05-27 17:30:17 EDT ---

Since that was a scratch build, I do not have the sources from pkgs cvs to
reproduce your build problem.

--- Additional comment from Orion Poplawski on 2008-05-27 17:36:54 EDT ---

http://www.cora.nwra.com/~orion/fedora/gridengine-6.2-0.beta.1.fc10.src.rpm

--- Additional comment from Roland McGrath on 2008-05-27 18:45:27 EDT ---

source/clients/qmon/qmon_cplx.c: 61: #include "spool/flatfile//sge_flatfile_obj.h"
source/libs/spool/flatfile/test_sge_spooling_flatfile.c: 47: #include
"spool/flatfile//sge_flatfile.h"
source/libs/spool/flatfile/test_sge_spooling_flatfile.c: 50: #include
"spool/flatfile//sge_flatfile_obj.h"


--- Additional comment from Orion Poplawski on 2008-05-28 11:10:22 EDT ---

Thanks, didn't think to check for includes inside the source files.

--- Additional comment from Bug Zapper on 2009-06-09 18:52:05 EDT ---


This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

--- Additional comment from Bug Zapper on 2009-07-14 14:16:29 EDT ---


Fedora 9 changed to end-of-life (EOL) status on 2009-07-10. Fedora 9 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

--- Additional comment from Martin Stransky on 2013-01-08 13:34:38 EST ---

It affects Firefox 18 build in latest Fedoras.

--- Additional comment from Martin Stransky on 2013-01-08 16:35:11 EST ---

Unfortunately this bug blocks Firefox security updates in all Fedoras.

--- Additional comment from Panu Matilainen on 2013-01-09 03:19:41 EST ---

Well its a stray "/" added somewhere between Firefox 17 and 18, either in the sources or the build process. Find and fix that and it'll build.

--- Additional comment from Martin Stransky on 2013-01-09 03:24:59 EST ---

Yes, I understand the mechanism. But the sources and changes are massive, there are new libraries imported and so. 

The affected chars "//" are not present in build logs, it's somewhere in include paths, which is extremely difficult to find and fix (the Makefiles are generated by scripts at build start so I can't patch them directly).

--- Additional comment from Martin Stransky on 2013-01-09 03:29:09 EST ---

As a workaround I'm building Firefox for this update without debuginfo packages due to this bug.

--- Additional comment from Jakub Jelinek on 2013-01-09 03:32:46 EST ---

Just run readelf -wl on the binaries and libraries before debugedit is run on them, and look for //.  That will show you in which binary or shared library it is, and on which include file (or main source file) it was.

--- Additional comment from Martin Stransky on 2013-01-09 03:37:37 EST ---

Jakub, thanks, I have such list...but it's a big list and as I told here the Makefiles are generated by python scripts and the include path is composed from many components, like "parent_dir\\..\some_dir" and so.

I wonder is it really so difficult to fix this bug in rpm?

--- Additional comment from Panu Matilainen on 2013-01-09 05:32:44 EST ---

See comment #2 for explanation of the underlying issues.

Suse has carried a patch that's supposed to address the issue but it causes "badly formed extended line op encountered!" warnings from readelf on the resulting .debug files so I'm not very convinced.

--- Additional comment from Jakub Jelinek on 2013-01-09 05:43:48 EST ---

If there are readers that assume that right after the directory and filename table is the line program, then some cases are impossible to handle on the debugedit side, unless we turn it into a full blown dwarf rewriter (could reuse parts of dwz code for it which is also a dwarf rewriter, but it would be a lot of work).  If we stay with the current way of doing things, then perhaps debugedit.c could handle some cases (where there are any / characters in the directory table other than at the beginning of the paths (POSIX makes //path different from /path, while /path/something and /path//something should be equivalent), by doubling one of the non-initial slashes somewhere, thus growing the directory table again by 1 character (basically find the right spot after dir, then memmove everything after it by one character later and insert '/').

debugedit.c inserts apparently not just one padding string, but up to two:

      if (shrank > 0)
        {
          if (--shrank == 0)
            error (EXIT_FAILURE, 0,
                   "canonicalization unexpectedly shrank by one character");
          else
            {
              memset (ptr, 'X', shrank);
              ptr += shrank;
              *ptr++ = '\0';
            }
        }

      if (abs_dir_cnt + abs_file_cnt != 0)
        {
          size_t len = (abs_dir_cnt + abs_file_cnt) * (base_len - dest_len);

          if (len == 1)
            error (EXIT_FAILURE, 0, "-b arg has to be either the same length as -d arg, or more than 1 char longer");
          memset (ptr, 'X', len - 1);
          ptr += len - 1;
          *ptr++ = '\0';
        }

The uses of abs_dir_cnt in there really confuses me, I'd have expected that the directory shrinking was already accounted for in the shrank length, but I admit I haven't tried to compile this now and debug on testcases.

Anyway, supposedly the two could have been also merged into one padding string if both are needed (two padding string need to be at least 4 bytes long, while one could be just 2 bytes long or longer.

--- Additional comment from Martin Stransky on 2013-01-10 08:55:16 EST ---

I managed to fix the Firefox Makefile generator so it's no longer critical...but would be great to solve it somehow.

--- Additional comment from  on 2013-02-23 22:54:32 EST ---

FWIW, I had the same problem building Oracle enabled Zabbix.  My problem turned out to be the ORACLE_HOME and ORACLE_BASE environment variables.  

This causes the error:

ORACLE_BASE=/u01/app/oracle/product/11.2.0.3/client/
ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/client/

This does not:

ORACLE_BASE=/u01/app/oracle/product/11.2.0.3/client
ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/client

--- Additional comment from Eric Smith on 2013-02-26 13:46:35 EST ---

It would be really useful if debugedit actually output the path that it had trouble with, rather than leaving us guessing.  I'm having this trouble with updating the Meshlab package to 1.3.2.  I might try to patch debugedit if I can't find the problem easily.

--- Additional comment from Miro Hrončok on 2013-02-27 03:37:40 EST ---

(In reply to comment #26)
> It would be really useful if debugedit actually output the path that it had
> trouble with, rather than leaving us guessing.  I'm having this trouble with
> updating the Meshlab package to 1.3.2.  I might try to patch debugedit if I
> can't find the problem easily.

See comment #20
> Just run readelf -wl on the binaries and libraries before debugedit is run
> on them, and look for //.  That will show you in which binary or shared
> library it is, and on which include file (or main source file) it was.

Might help.

--- Additional comment from Miro Hrončok on 2013-02-28 14:36:54 EST ---

(In reply to comment #20)
> Just run readelf -wl on the binaries and libraries before debugedit is run
> on them, and look for //.  That will show you in which binary or shared
> library it is, and on which include file (or main source file) it was.

Here's a very long build.log http://koji.fedoraproject.org/koji/taskinfo?taskID=5065189 (>100 MB).

readelf -wl was run on the libraries and grep // gives me nothing usefull. Any hints?

--- Additional comment from Fedora End Of Life on 2013-04-03 15:48:38 EDT ---

This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

--- Additional comment from Rex Dieter on 2014-12-12 20:23:48 EST ---

Fwiw, I'm seeing this recently on fedora 21 (but not fedora 20), trying to build a qtwebengine package, 

here's a link to 
$ readelf -wl /home/rdieter1/pkgs.fedoraproject.org/work/BUILDROOT/qt5-qtwebengine-5.4.0-1.fc21.x86_64/usr/lib64/libQt5WebEngineCore.so.5.4.0 
output:
https://rdieter.fedorapeople.org/rpms/qt5/qtwebengine-debuginfo.log.xz

One thing that jumps out are some errors from readelf:
readelf: Warning: badly formed extended line op encountered!
(repeated several times)

--- Additional comment from Rex Dieter on 2014-12-12 20:35:25 EST ---

And build log:
https://rdieter.fedorapeople.org/rpms/qt5/qtwebengine-build.log

--- Additional comment from Fedora End Of Life on 2015-11-04 10:25:53 EST ---

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.