Bug 450700

Summary: Ccache should not modify default path
Product: [Fedora] Fedora Reporter: Jonathan S. Shapiro <shap>
Component: ccacheAssignee: Ville Skyttä <ville.skytta>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 9CC: spetreolle
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-06-11 20:09:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jonathan S. Shapiro 2008-06-10 15:00:16 UTC
We have just tracked down a compile problem that was due to ccache. The ccache
cache was stale due to a compiler upgrade. The user was not intending to use
ccache at all. Because ccache modifies the system default path, they did not
realize they were doing so.

Ccache is good and almost always right, but it is a power tool. Users who want
it are amply competent to modify their own paths. The CCache RPM should not
modify the systemwide path.

I am marking this "high" because it can impact all manner of builds where a
compiler has been upgraded, including those where the compiler upgrade and the
associated recompile are being performed for security reasons.

This is also an issue in older Fedora releases, notably FC8.

Comment 1 Ville Skyttä 2008-06-10 18:16:14 UTC
I disagree.  Users who want ccache are also competent to build it from source. 
The whole point of the Fedora ccache package is to make it as easy as possible
to use it.

/etc/profile.d/ccache*.sh is a config file, it can be edited if the local
sysadmin  chooses not to enable it for all users.  It also contains instructions
how individual users can disable it.

Comment 2 Jonathan S. Shapiro 2008-06-10 19:09:03 UTC
This user did NOT want ccache. That is the point. The problem is that the
machine was not a single user machine, and the installation of ccache is
"default enable". The situation today is that if any user wants ccache, all
other users are required to take action to suppress ccache.

Users who want ccache can simply add it to their path. The fact that one user
wants to use it should not alter the behavior seen by other users -- and
*certainly* not in a way that causes compiles to break!

I note that you also have not addressed the security aspects of this, so I
respectfully suggest that closing this bug is premature.

Comment 3 Jonathan S. Shapiro 2008-06-10 19:09:57 UTC
Also, you have failed to address the fact that the mere fact of ccache being
installed on the machine caused compiles to fail for a user who did not wish to
execute ccache.

Comment 4 Ville Skyttä 2008-06-10 22:02:28 UTC
(In reply to comment #2)
> The problem is that the machine was not a single user machine, and the 
> installation of ccache is "default enable".

If by this you mean that the package is installed by default in Fedora, that's
not my call.  Unless I'm mistaken, Fedora release engineering or steering
committee governs such things.

If you mean that the ccache package does auto-enable itself after installation,
as said, that's intentional.  If someone installs new packages, in my opinion it
is the responsibility of that individual to find out what (if any) system wide
effects it has, and to communicate the opt-in/opt-out (depending how he chooses
to configure the software or not) possibilities to others so they can make
informed choices.

> The situation today is that if any user wants ccache, all other users are
> required to take action to suppress ccache.

That's incorrect, see last paragraph of comment 1.  (For example something like
"CCACHE_DISABLE=1" in /etc/profile.d/ccache.sh, then "unset CCACHE_DISABLE" in
~/.bash_profile for ones who want to opt in.)

> I note that you also have not addressed the security aspects of this

If someone does "production" builds, especially security related ones in a setup
that has unknown/unintentional things installed (such as I gather ccache was in
this case), that setup has really severe problems.  Nothing that could be done
to the ccache package would change that.

(In reply to comment #3)
> Also, you have failed to address the fact that the mere fact of ccache being
> installed on the machine caused compiles to fail

That sounds like a different bug report, there's not enough information in this
one to say much more about it.  Moreover, it sounds more like an upstream one
than a packaging issue.  But FWIW, ccache does include the timestamp, full path,
and size of the actual compiler executables invoked in its hashes which should
protect against most usual compiler change scenarios.  That won't obviously fix
everything though, which I'm sure is one of the reasons the CCACHE_DISABLE
environment variable exists.

My personal opinion is unchanged, I still don't consider auto-enabling being a
bad thing in this particular case, even if there are some rare cases where it
could cause confusion because it's really easy to opt out individually or on
system level.  Until there's clear community consensus that says otherwise, or
more compelling evidence that the current choice is the wrong one, I'm keeping
the setup as is.

Comment 5 Jonathan S. Shapiro 2008-06-11 01:23:43 UTC
We still aren't communicating.

No, the package isn't installed by default in Fedora. In this case I happen to
know that ccache was not installed intentionally by any user. It was installed
to satisfy a dependency.

Yes, ccache does various things with hashes to try to ensure that it does the
right thing. Unfortunately, ccache does not include any hash of the compiler,
linker, or other tools in its computation. In consequence, if the compiler is
updated, and the new compiler is used to compile the same (identical) input,
ccache will substitute the output from the *old* compiler, because it does not
detect the compiler update. This failure mode for ccache is 100% reproducible.
We have seen it lots of times in our use of ccache over the years.

Your position about the responsibilities of installers is unrealistic. It is
flatly impossible for the person installing packages to even begin to understand
the interactions of all of the packages that they are installing. I just
downgraded a machine from F9 because of instability issues. The off-the-DVD
install with software development packages enabled installed 1250+ packages. If
you like, I will send you the list. Unless you as a Fedora developer (and
therefore in the best position to know), can credibly claim that you understand
what all 1250+ of those packages do, I think that your position on "responsible
installation" is untenable. It's a "blame the user" (or in this case, the
administrator) response. The responsibility for sensible defaults lies with the
package developer first and then with the packager.

I do agree that the misbehavior could be avoided if CCACHE_DISABLE=1 is added to
/etc/profile.c/ccache.sh, and I agree that this would have resolved the
particular user's problem. I do not agree that this is a good solution, as it
creates a burden for people who DO use ccache. Basically, your proposed "naive
user safe" installation places a burden on both knowledgeable users (who must
unset the variable) and also on adminstrators (who must edit
/etc/profile.d/ccache.sh). It is better in all respects to simply have
knowledgeable developers add the path fragment for themselves. The burden on the
knowledgeable user is the same as your proposal, but the administrator isn't
called on to repair a bad deployment decision.

I would be happy with either of two outcomes:

 1. Fix ccache so that it handles tool changes correctly. That is: so that it
    generates correct output in all cases.

 2. Fix the ccache install so that installing ccache does not break builds
    for innocent bystanders.

What I feel very strongly about is that *one* of these outcomes is mandatory
(you pick), but until ccache is fully correctness-preserving it should not
install in default-enabled form.

By fully correctness preserving, I do not mean to worry about hash collisions.
The heat death of the universe will happen sooner, but heat death is a low
priority bug that has to be fixed upstream. :-)

Comment 6 Ville Skyttä 2008-06-11 20:09:40 UTC
This is a judgement call between convenience and rare breakage (2nd bug report
in 5 years about it, and first that asks to be made opt-in instead of opt-out).
 I don't know how to put my opinion more clearly than I already have, so I'm
closing this NOTABUG again.  Please discuss on fedora-devel mailing list if you
feel strongly about it.  There's no need to repeat the same arguments here, but
as said, if there's clear consensus that the package should not default-enable
itself, something will happen to it.

Just a couple of notes/questions:

> It was installed to satisfy a dependency.

I'm interested in hearing which dependency this is.  "repoquery --whatrequires
--alldeps ccache" on my F-8 (+ various other repositories) system does not list
anything.  I'm unable to try later distro versions at the moment.

> ccache does not include any hash of the compiler

As said in my previous comment, it DOES include the compiler path, size and time
stamp in its hash routines, see find_hash() in ccache.c.  And also as said, and
as the comments in ccache.c also say, yes, I'm fully aware that this isn't
completely reliable even against the compiler, let alone other things the
compiler might invoke internally.