Bug 759592 - Debug info is broken - use CCACHE_HASHDIR
Summary: Debug info is broken - use CCACHE_HASHDIR
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: ccache
Version: 16
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Ville Skyttä
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-12-02 19:19 UTC by Jan Kratochvil
Modified: 2012-01-31 23:54 UTC (History)
3 users (show)

Fixed In Version: ccache-2.4-20.el5
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-01-31 23:54:41 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Fix - enable CCACHE_HASHDIR. (587 bytes, patch)
2011-12-02 19:19 UTC, Jan Kratochvil
no flags Details | Diff

Description Jan Kratochvil 2011-12-02 19:19:33 UTC
Created attachment 539743 [details]
Fix - enable CCACHE_HASHDIR.

Description of problem:
I was debugging now GDB as a GDB developer to find out the breakage is due to ccache.

Version-Release number of selected component (if applicable):
ccache-3.1.6-1.fc16.x86_64

How reproducible:
Always.

Steps to Reproduce:
rm -rf a b;mkdir a;echo 'int main(void){return 0;}' >a/f.c;cp -a a b;(cd a;gcc -c -o f.o f.c -Wall -g);(cd b;gcc -c -o f.o f.c -Wall -g);readelf -wi b/f.o|grep DW_AT_comp_dir

Actual results:
   <15>   DW_AT_comp_dir    : (indirect string, offset: 0x56): /tmp/ccachebug/a

Expected results:
   <15>   DW_AT_comp_dir    : (indirect string, offset: 0x56): /tmp/ccachebug/b

Additional info:
Fedora (and Red Hat...) is working and achieved various improvements of debug infos (VTA, entryval and others), incl. ABRT project for automating the debugging process.  Producing broken debug info incl. in Koji is just not worth the performance.

There can be some note in rpm %description about CCACHE_HASHDIR performance benefits but just installing a new package really must not break system functionality in any way.

I was debugging here false GDB testsuite failure on:
Running ./gdb.trace/change-loc.exp ...
FAIL: gdb.trace/change-loc.exp: 1 trace: tracepoint with two locations
FAIL: gdb.trace/change-loc.exp: 1 trace: tracepoint with three locations
FAIL: gdb.trace/change-loc.exp: 1 trace: tracepoint with two locations (unload)
FAIL: gdb.trace/change-loc.exp: 1 trace: tfind frame 0
FAIL: gdb.trace/change-loc.exp: 2 trace: tracepoint with two locations
FAIL: gdb.trace/change-loc.exp: 2 trace: tracepoint with three locations
FAIL: gdb.trace/change-loc.exp: 2 trace: tracepoint with two locations (unload)
FAIL: gdb.trace/change-loc.exp: 2 trace: tfind frame 1
FAIL: gdb.trace/change-loc.exp: 2 trace: tfind frame 2
FAIL: gdb.trace/change-loc.exp: 1 ftrace: tracepoint with two locations
FAIL: gdb.trace/change-loc.exp: 1 ftrace: tracepoint with three locations
FAIL: gdb.trace/change-loc.exp: 1 ftrace: tracepoint with two locations (unload)
FAIL: gdb.trace/change-loc.exp: 1 ftrace: tfind frame 0
FAIL: gdb.trace/change-loc.exp: 2 ftrace: tracepoint with two locations
FAIL: gdb.trace/change-loc.exp: 2 ftrace: tracepoint with three locations
FAIL: gdb.trace/change-loc.exp: 2 ftrace: tracepoint with two locations (unload)
FAIL: gdb.trace/change-loc.exp: 2 ftrace: tfind frame 1
FAIL: gdb.trace/change-loc.exp: 2 ftrace: tfind frame 2

Comment 1 Ville Skyttä 2011-12-03 13:57:11 UTC
CCACHE_HASHDIR set sounds like a fine cautious default.  I'll look into it, but I'd rather get bug 709561 sorted out in the same update as this.

Comment 2 Fedora Update System 2011-12-04 21:12:45 UTC
ccache-3.1.4-5.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/ccache-3.1.4-5.fc15

Comment 3 Ville Skyttä 2011-12-04 21:14:23 UTC
F-15 is the only distro for which things are clear wrt bug 709561 so update submitted only for it so far, the rest will follow later.

Comment 4 Fedora Update System 2011-12-13 21:57:34 UTC
ccache-3.1.4-5.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 5 Fedora Update System 2011-12-17 22:28:03 UTC
ccache-3.1.6-2.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/ccache-3.1.6-2.el6

Comment 6 Joel Rosdahl 2011-12-18 10:32:54 UTC
The proposed patch makes ccache essentially useless for many scenarios where you want to rebuild something in another directory. I think that it should be up to the user to decide whether CCACHE_HASHDIR should be enabled or not, and the formulation "Broken debug info is not acceptable" does not help with that. An improved comment could be something along the lines of "When using GCC 4.6 [or whatever the version is], CCACHE_HASHDIR needs to be set in order to produce correct debug information at all times. If you don't care about this, or use another compiler, then disabling CCACHE_HASHDIR (possibly in combination with enabling CCACHE_BASEDIR) makes it possible for ccache to produce cache hits regardless of build directory.".

Furthermore, I totally agree that just installing a new package must not break system functionality. Therefore, my opinion is that it's a bad idea to enable ccache automatically when installing the ccache package. I think that it should be up to the user to make an informed choice to enable ccache, so that the responsibility of setting it up correctly is shifted to the user. ccache has too many known and unknown ways of potentially altering the build result in edge cases.

Comment 7 Jan Kratochvil 2011-12-18 10:45:50 UTC
(In reply to comment #6)
> The proposed patch makes ccache essentially useless for many scenarios where
> you want to rebuild something in another directory.

TBH I do not understand too much when it can be useful.  I use a single directory with ~40 GIT branches of GDB and I switch the work between these branches in that single directory.  While one can switch to a different directory during one work session involving many rebuilds one stays in a single directory.


> Therefore, my opinion is that it's a bad idea to enable
> ccache automatically when installing the ccache package.

ccache with CCACHE_HASHDIR brings 1m24s->0m44s speedup for me while it is (should be) transparent.  I do not see much reasons why not to enable it.


> it should be up to the user to make an informed choice to enable ccache,

But for example the new 1GB ~/.ccache directory may be a serious enough issue the manual ccache enable may make sense.  Personally I find the automatic enable with CCACHE_HASHDIR OK, though.


> ccache has too many known and unknown ways of potentially altering the build
> result in edge cases.

If there are other non-transparencies those are new Bugs which should be fixed.

Comment 8 Joel Rosdahl 2011-12-18 15:25:41 UTC
(In reply to comment #7)
> TBH I do not understand too much when it can be useful. I use a single
> directory with ~40 GIT branches of GDB and I switch the work between these
> branches in that single directory. While one can switch to a different
> directory during one work session involving many rebuilds one stays in a
> single directory.

Yes, that's indeed a valid use case, but not at all the only one. Examples of
others:

* You use Subversion and keep several branches checked out in different
  directories.
* You have a server where a ccache is shared between multiple users, each using
  build directories of their own.
* You have a build server that checks out a new copy of the source code in a
  temporary directory (with a different name each time) to build and test your
  project automatically each night or after each commit or so.

I'd say that many seldom or never use ccache like you do.

> ccache with CCACHE_HASHDIR brings 1m24s->0m44s speedup for me while it is
> (should be) transparent. I do not see much reasons why not to enable it.

Often transparent, yes, but limits the scenarios when ccache is useful, as
discussed above.

> If there are other non-transparencies those are new Bugs which should be
> fixed.

There are non-transparencies that simply can't be fixed, at least not without
major effort. One example:
<http://www.mail-archive.com/ccache@lists.samba.org/msg00382.html>.

Comment 9 Jan Kratochvil 2011-12-18 15:57:15 UTC
(In reply to comment #8)
> There are non-transparencies
> One example:
> <http://www.mail-archive.com/ccache@lists.samba.org/msg00382.html>.

In such case this is a Bug and ccache really should not be enabled by default until it is fixed.


> that simply can't be fixed, at least not without major effort.

What's wrong with always forcing --MD?

$ cat 1.S
.incbin "binary"
$ as -o 1.o --MD 1.md 1.S
$ cat 1.md 
1.o: binary 1.S

Comment 10 Joel Rosdahl 2011-12-18 16:30:08 UTC
(In reply to comment #9)
> In such case this is a Bug and ccache really should not be enabled by default
> until it is fixed.

I agree. And to me, it doesn't feel reasonable to switch the default between
enabled and disabled depending on whether ccache has known bugs or only unknown
bugs.

> What's wrong with always forcing --MD?
>
> $ cat 1.S
> .incbin "binary"
> $ as -o 1.o --MD 1.md 1.S
> $ cat 1.md
> 1.o: binary 1.S

I don't quite understand what you're implying. Do you mean that ccache should
silently pass --MD to something (what? ccache doesn't call as directly) and then parse the resulting .md file to find out the .incbin files?

Two other wrongs:

1. As far as I can tell, --MD is undocumented.
2. --MD is likely not supported by non-GNU compilers/assemblers.

-- Joel

Comment 11 Jan Kratochvil 2011-12-18 16:42:54 UTC
(In reply to comment #10)
> I agree. And to me, it doesn't feel reasonable to switch the default between
> enabled and disabled depending on whether ccache has known bugs or only
> unknown bugs.

It depends to some level.  GCC also sometimes has known bugs, being in the compiler toolchain is sensitive.


> I don't quite understand what you're implying. Do you mean that ccache should
> silently pass --MD to something (what? ccache doesn't call as directly)

ccache probably calls always gcc, which is right.  You can always pass as flags by gcc -Wa,--MD,1.md.  For --MD maybe you could use some other GCC options like -M, -MF etc. but I do not see now which ones.


> and then parse the resulting .md file to find out the .incbin files?

Yes.


> 1. As far as I can tell, --MD is undocumented.

$ as --help |grep MD
  --MD FILE               write dependency information in FILE (default none)
$ info '(as)MD'


> 2. --MD is likely not supported by non-GNU compilers/assemblers.

If we never use the GNU extensions, does GNU make any sense?  Sure make it an optional extension/fix.

Comment 12 Ville Skyttä 2011-12-18 17:09:14 UTC
Please note that ccache is not installed by default in Fedora.  And apart from things that absolutely require manual configuration or are network facing daemons, installing stuff in Fedora implies enabling it.  I think you're overestimating the difficulty of "yum remove ccache" or setting CCACHE_DISABLE or removing the ccache symlink dir from $PATH.

Please also note that ccache has been auto-enabled in Fedora (if installed) for 5+ years, with very few bug reports.  If being bug free would really be a requirement for something to be usable without an opt-in, there's very little one could do with a fresh installation out of the box.

For released distro versions, with the information at hand, I will not consider shipping a ccache update that suddenly flips it to disabled.  Also, it's not at all clear that I will be doing that for future distro versions either as long as ccache is not installed by default.

I understand the limitations of CCACHE_HASHDIR but I think it's a reasonable default -- do as much right as you can out of the box, and provide switches to get more performance out of it for users who can live with the consequences.

Even though the incbin discussion is good education, please keep the discussion here relevant to its original subject, and open new bugs for other things if necessary, thank you.

Comment 13 Jan Kratochvil 2011-12-18 17:15:04 UTC
(In reply to comment #12)
> I think you're overestimating the difficulty of "yum remove ccache" or
> setting CCACHE_DISABLE or removing the ccache symlink dir from $PATH.

The difficulty is finding the cause of the broken build.  I am used to investigate GCC bugs and debug GDB, I was deep in GDB debugging for ... 1 hour? before I found it is caused by ccache this time.  I may be stupid and I could find it earlier but I just expected it is another GDB bug to fix that day.

Sure as the toolchain engineer I am biased, I am not a regular user.
And besides all I also do not agree with various other Fedora decisions.
Thanks for all the work for Fedora.

Comment 14 Joel Rosdahl 2011-12-18 17:52:25 UTC
(In reply to comment #11)
> ccache probably calls always gcc, which is right.  You can always pass as flags
> by gcc -Wa,--MD,1.md.

Thanks. That could indeed be done if ccache had a "compiler feature detection"
mechanism.

> > 1. As far as I can tell, --MD is undocumented.
>
> $ as --help |grep MD
>   --MD FILE               write dependency information in FILE (default none)
> $ info '(as)MD'

OK, good. It's missing from the man page, though, which is what I read.

> Sure make it an optional extension/fix.

That's currently not easily done.

Comment 15 Jan Kratochvil 2011-12-18 18:12:28 UTC
(In reply to comment #14)
> OK, good. It's missing from the man page, though, which is what I read.

GNU Project does not use man pages.
  http://www.gnu.org/prep/standards/html_node/Man-Pages.html

Comment 16 Joel Rosdahl 2011-12-18 18:22:48 UTC
(In reply to comment #12)
> Please note that ccache is not installed by default in Fedora. And apart from
> things that absolutely require manual configuration or are network facing
> daemons, installing stuff in Fedora implies enabling it. I think you're
> overestimating the difficulty of "yum remove ccache" or setting
> CCACHE_DISABLE or removing the ccache symlink dir from $PATH.

Sure, I don't think it's difficult to disable it, it's just that I think that
automatically enabling it doesn't adhere to the principle of least surprise.
But then again, I'm not at all a common user.

> If being bug free would really be a requirement for something to be usable
> without an opt-in, there's very little one could do with a fresh installation
> out of the box.

I don't quite agree with that analogy. Without automatic enabling, it is still
usable. My view is that ccache is more of an optional tool than a service that
gets enabled.

> For released distro versions, with the information at hand, I will not
> consider shipping a ccache update that suddenly flips it to disabled. Also,
> it's not at all clear that I will be doing that for future distro versions
> either as long as ccache is not installed by default.

I agree that keeping ccache enabled by default makes perfect sense from a
package maintainer's point of view. I just felt that I should make my opinion
on this heard, which is something I have wanted to do several times before but
for some reason didn't. Sorry about the noise. :-) I'll not push this anymore
in the future.

> I understand the limitations of CCACHE_HASHDIR but I think it's a reasonable
> default -- do as much right as you can out of the box, and provide switches
> to get more performance out of it for users who can live with the
> consequences.

In that case I think you should disable the direct mode as well by default,
unfortunately. See <https://bugzilla.samba.org/show_bug.cgi?id=8424>. I think I
will do the same in ccache 3.2, and I'll probably also make CCACHE_HASHDIR the
default as well to be conservative.

> Even though the incbin discussion is good education, please keep the
> discussion here relevant to its original subject, and open new bugs for other
> things if necessary, thank you.

So, to keep to the topic, what do you think about my suggested comment? I think
that a better explaination of why CCACHE_HASHDIR is set by default would be
good in this case, since it's a change of ccache's own defaults.

Comment 17 Fedora Update System 2012-01-02 19:53:43 UTC
ccache-3.1.6-2.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2012-01-08 20:41:57 UTC
ccache-3.1.7-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/ccache-3.1.7-1.fc16

Comment 19 Fedora Update System 2012-01-08 21:06:50 UTC
ccache-2.4-20.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/ccache-2.4-20.el5

Comment 20 Fedora Update System 2012-01-11 08:00:08 UTC
Package ccache-2.4-20.el5:
* should fix your issue,
* was pushed to the Fedora EPEL 5 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing ccache-2.4-20.el5'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2012-0066/ccache-2.4-20.el5
then log in and leave karma (feedback).

Comment 21 Fedora Update System 2012-01-19 01:31:06 UTC
ccache-3.1.7-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 22 Fedora Update System 2012-01-31 23:54:41 UTC
ccache-2.4-20.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.