Bug 392851

Summary: spamassassin doesn't detect its own sample-spam as spam
Product: [Fedora] Fedora Reporter: Bertil Askelid <bertil>
Component: spamassassinAssignee: Warren Togami <wtogami>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: low    
Version: 8CC: bugzilla
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-12-16 14:09:58 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
spamassassin -D output none

Description Bertil Askelid 2007-11-20 19:12:51 UTC
Description of problem:

Run spamc on spamassassin sample spam, not detected as spam

Version-Release number of selected component (if applicable):

spamassassin-3.2.3-44.fc8

How reproducible:

always

Steps to Reproduce:
1.spamc -f < /usr/share/doc/spamassassin-3.2.3/sample-spam.txt
2.
3.
  
Actual results:

X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on z.lan
Subject: Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender>
To: Recipient <recipient>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

This is the GTUBE, the
	Generic
	Test for
	Unsolicited
	Bulk
	Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.


Expected results:

As run on FC7.

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on
	*********
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=1002.5 required=5.0 tests=GTUBE,NO_RECEIVED,
	NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,RAZOR2_CHECK
	autolearn=no version=3.2.3
X-Spam-Report: 
	* -0.0 NO_RELAYS Informational: message was not relayed via SMTP
	* 1000 GTUBE BODY: Generic Test for Unsolicited Bulk Email
	*  0.5 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
	*  1.5 RAZOR2_CF_RANGE_E4_51_100 Razor2 gives engine 4 confidence level
	*      above 50%
	*      [cf: 100]
	*  0.5 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
	*      [cf: 100]
	* -0.0 NO_RECEIVED Informational: message has no Received headers
Subject: [SPAM] Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender>
To: Recipient <recipient>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Prev-Subject: Test spam mail (GTUBE)

This is the GTUBE, the
	Generic
	Test for
	Unsolicited
	Bulk
	Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.


Additional info:

/etc/mail/spamassassin/local.cf

# These values can be overridden by editing ~/.spamassassin/user_prefs.cf 
# (see spamassassin(1) for details)

# These should be safe assumptions and allow for simple visual sifting
# without risking lost emails.

required_score 5
report_safe 0

~/.spamassassin/user_prefs

no overriding info

Comment 1 Warren Togami 2007-11-20 19:29:52 UTC
> spamc -f < /usr/share/doc/spamassassin-3.2.3/sample-spam.txt

What is -f supposed to do?  I don't see -f in spamc's man page.

Furthermore, trying this on my system works just fine both with and without -f.

Comment 2 Bertil Askelid 2007-11-20 19:57:48 UTC
You are correct. -f is not needed. I ran spamc without -f on both my FC8 and my
FC7 machines. Same result as before. Fails to detect it as a spam on FC8, while
FC7 is just fine.

I've just installed FC8 from the DVD and then done yum from Fedora, Livnia,
ATrpms, and Freshrpms to update my system with the latest.

As it works for you, there must be something that collides with spamassassin on
my system.  Please advise me on how to debug spamassassin to figure out and correct.

Thanks!

Comment 3 Bertil Askelid 2007-11-21 02:38:44 UTC
Downgraded to the spamassassin-3.2.0-1.fc7 from the FC7 release DVD. Works just
fine on my FC8 system.


~> spamc < /usr/share/doc/spamassassin-3.2.0/sample-spam.txt
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on z.lan
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=999.9 required=5.0 tests=BAYES_00,GTUBE,NO_RECEIVED,
	NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,RAZOR2_CHECK
	autolearn=no version=3.2.0
X-Spam-Report: 
	* -0.0 NO_RELAYS Informational: message was not relayed via SMTP
	* 1000 GTUBE BODY: Generic Test for Unsolicited Bulk Email
	* -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
	*      [score: 0.0000]
	*  1.5 RAZOR2_CF_RANGE_E4_51_100 Razor2 gives engine 4 confidence level
	*      above 50%
	*      [cf: 100]
	*  0.5 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
	*  0.5 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
	*      [cf: 100]
	* -0.0 NO_RECEIVED Informational: message has no Received headers
Subject: [SPAM] Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender>
To: Recipient <recipient>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Prev-Subject: Test spam mail (GTUBE)

This is the GTUBE, the
	Generic
	Test for
	Unsolicited
	Bulk
	Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.



Comment 4 Warren Togami 2007-11-21 02:46:16 UTC
You mention AtRPMS.  AtRPMS is effectively a fork of Fedora because it replaces
system packages.  Please see if you can reproduce this bug with stock Fedora 8
packages.  Nobody else is able to reproduce it with stock Fedora 8.

If an AtRPMS package is causing this failure, please report the bug to that project.


Comment 5 Axel Thimm 2007-11-23 07:32:15 UTC
> You mention AtRPMS.  AtRPMS is effectively a fork of Fedora because it
> replaces system packages.

Off topic to the actual bug report, but that's not really a sane conclusion.
Even fedora.us from where Waren stems from replaced system packages even more
hard core packages like shadow-utils. Was fedora.us a fork of Red Hat Linux? And
every other repo out there had to do the same in similar situation including
dag, freshrpms, livna etc. replacing rpm itself or mp3-castrated software. So
the first one not having had to replace a RHL or Fedora package should throw the
first stone.

ATrpms is a generic 3rd party add-on repo improving and enhancing your Fedora
and/or RHEL experience. If ATrpms ever wants to fork something you'll know.

Back on-topic: Bertil you need to make sure you actually have the spamassassin
service running. Like "service spamassassin start". Since you have the same
issue with Fedora 8's plain and ATrpms' enhanced spamassassin packages, but not
FC7's, it looks like either a non-started service or a misconfigration
somewhere. It would be strange if the F8 plain spamassassin package would be
confused by presence of other helper software and F7's not (but nothing can be
outruled).

Since you filed a bug over at ATrpms we can follow up there in case you find out
that there really is an issue with ATrpms' packages after having restarted the
spamassassin service (spamd). But I wouldn't be getting any mails if the
spamassassin & support packages at ATrpms were that broken. ;)


Comment 6 Warren Togami 2007-11-24 00:13:03 UTC
> Off topic to the actual bug report, but that's not really a sane conclusion.
> Even fedora.us from where Waren stems from replaced system packages even more
> hard core packages like shadow-utils.

Give me a break.  shadow-utils in fedora.us contained a one-liner patch to fix a
segfault crasher.  fedora.us replaced so few things, and when it did it did in
some insignificant way.

Comment 7 Bertil Askelid 2007-11-24 17:30:11 UTC
Steps taken to investigate bug. This shows that the spamd server indeed is
running while the bug happens. As you can see, the FC8 versions of spamassassin
from ATrpms (spamassassin-3.2.3-44.fc8.i386.rpm) and Fedora
(spamassassin-3.2.3-2.fc8.i386.rpm) both fail while the one from FC7
(spamassassin-3.2.0-1.fc7.i386.rpm) works just fine.

bertil| rpm -e spamassassin
bertil| rpm -ivh spamassassin-3.2.3-44.fc8.i386.rpm
Preparing...                ########################################### [100%]
   1:spamassassin           ########################################### [100%]
bertil| /etc/rc.d/init.d/spamassassin start
Starting spamd: [  OK  ]
bertil| spamc < /usr/share/doc/spamassassin-*/sample-spam.txt
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on z.lan
Subject: Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender>
To: Recipient <recipient>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

This is the GTUBE, the
	Generic
	Test for
	Unsolicited
	Bulk
	Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.

bertil| rpm -e spamassassin
bertil| rpm -ivh spamassassin-3.2.3-2.fc8.i386.rpm
Preparing...                ########################################### [100%]
   1:spamassassin           ########################################### [100%]
bertil| /etc/rc.d/init.d/spamassassin start
Starting spamd: [  OK  ]
bertil| spamc < /usr/share/doc/spamassassin-*/sample-spam.txt
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on z.lan
Subject: Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender>
To: Recipient <recipient>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

This is the GTUBE, the
	Generic
	Test for
	Unsolicited
	Bulk
	Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.

bertil| rpm -e spamassassin
bertil| rpm -ivh spamassassin-3.2.0-1.fc7.i386.rpm
Preparing...                ########################################### [100%]
   1:spamassassin           ########################################### [100%]
bertil| /etc/rc.d/init.d/spamassassin start
Starting spamd: [  OK  ]
bertil| spamc < /usr/share/doc/spamassassin-*/sample-spam.txt
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on z.lan
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=1002.5 required=5.0 tests=GTUBE,NO_RECEIVED,
	NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,RAZOR2_CHECK
	autolearn=no version=3.2.0
X-Spam-Report: 
	* -0.0 NO_RELAYS Informational: message was not relayed via SMTP
	* 1000 GTUBE BODY: Generic Test for Unsolicited Bulk Email
	*  1.5 RAZOR2_CF_RANGE_E4_51_100 Razor2 gives engine 4 confidence level
	*      above 50%
	*      [cf: 100]
	*  0.5 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
	*  0.5 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
	*      [cf: 100]
	* -0.0 NO_RECEIVED Informational: message has no Received headers
Subject: Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender>
To: Recipient <recipient>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

This is the GTUBE, the
	Generic
	Test for
	Unsolicited
	Bulk
	Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.

bertil| 

Comment 8 Axel Thimm 2007-11-27 14:33:53 UTC
Bertil, in order to prove that ATrpms has nothing to do with this bug, please
just remove all mail related ATrpms packages and show that this still happens,
or just remove all ATrpms packages just to make it clear that the bug is in one
or several pure Fedora shipped packages w/o any 3rd party interaction.

At least I can't reproduce it on F8 i386/x86_64/ppc as well as several other
distros I'm using the same packaging.

If you continue posting examples about using ATrpms you give a convenient excuse
for non-ATrpms fans to look away.


Comment 9 Bertil Askelid 2007-11-28 04:10:00 UTC
I have removed *all* ATrpms and now been running my spam test as described above
on a *clean* Fedora Core 8 system. Same result as reported before. Only
spamassassin-3.2.0-1.fc7.i386.rpm detects the spam example as spam.

Comment 10 Need Real Name 2007-12-13 05:39:55 UTC
Created attachment 286651 [details]
spamassassin -D output

Comment 11 Need Real Name 2007-12-13 05:42:22 UTC
I am having the same problem - spamassassin is COMPLETELY BROKEN on F8 -- all
mail passes through with zero spam score vs. my previous FC6 installation where
it caught almost all spam.

IMHO, this blaming stuff on ATrpms is a red herring. There is DEFINITELY
something majorly screwed with spamassasin on F8 (or a conflict with some other
setup configuration).

Basically all mail gives a zero score.
I have attached the debug output from running on
/usr/share/doc/spamassassin-3.2.3/sample-spam.txt
(but unfortunately due to a type it precedes this comment)




Comment 12 Need Real Name 2007-12-14 15:53:10 UTC
Any suggestions on how to debug this?

Comment 13 Need Real Name 2007-12-17 06:48:23 UTC
OK I THINK I SOLVED THIS (after many wasted hours).

The following thread gave me the answer.
http://mail-archives.apache.org/mod_mbox/spamassassin-users/200605.mbox/%3c6bb609560605131018i2a51ff1ex42fe83fdee0eb4f8@mail.gmail.com%3e

Apparently spamassassin looks successively in the following directories for the
first non-empty directory that if present will override configuration data:
=item /var/lib/spamassassin/3.002003
=item /usr/share/spamassassin
=item /usr/share/spamassassin
=item /usr/local/share/spamassassin
=item /usr/share/spamassassin

(note /usr/share/spamassassin is repeated 3 times)

Well, I happen to have /var/lib/spamassassin/3.002003 on my new F8 system (but
no equivalent on my old F6 system).
This directory is populated by sa-update -- but apparently there is a bug that
sometimes the directory still is created even if sa-update fails to finish
downloading an update.

Now, sa-update is run daily at 4:10 using a crontab entry "hidden" in the
read-protected directory /etc/cron.d/ (not sure why it doesn't use cron.daily
like everything else).

Looking in my /var/log/sa-update.log file, I noticed the following error which
presumably is responsible for the faulty updating (and the incomplete creation
of the /var/lib/spamassassin/3.002003 directory) that causes spamassassin to choke:

error: GPG validation failed!
The update downloaded successfully, but it was not signed with a trusted GPG
key.  Instead, it was signed with the following keys:

    24F434CE 

Perhaps you need to import the channel's GPG key?  For example:

    wget http://spamassassin.apache.org/updates/GPG.KEY
    sa-update --import GPG.KEY

channel: GPG validation failed, channel failed

Now I'm not sure why I don't have the right GPG key.

(It's also not clear to me how this GPG key gets imported for anybody since I
don't see it being done in the rpm install scripts or being part of the rpm itself)

In any case, I was able to fix it by running:
sa-learn --import /usr/share/spamassassin/sa-update-pubkey.txt

Interestingly, I'm not sure why the following doesn't work (giving the same GPG
error):
sa-update --gpgpkey /usr/share/spamassassin/sa-update-pubkey.txt 
Note though that running 'sa-update --nogpg' did work.

So in summary, I conclude the following:
1. At least on my system, the GPG key was not imported properly. Not sure why
and not sure how many other people this affects
2. My problem where spamassassin broke completely was due to a combination of an
incomplete sa-update (which caused the creation of a partial 3.002003 directory)
plus the lack of a gpg key which prevented subsequent good sa-updates from
overwriting the complete one.

Hope this helps...

Comment 14 Bertil Askelid 2007-12-17 19:03:59 UTC
This *did* really help. Problem solved. Latest FC8 spamassassin works just fine.

So the install procedure in the spamassassin.rpm need to corrected to handle the
key setup.

Thanks!

Comment 15 Justin Mason 2007-12-17 20:21:47 UTC
Warren, could you open an upstream bug about this?  sa-update should not leave
the system in a state where there are no valid rules files, regardless of
whether the key was imported or not.

Comment 16 Need Real Name 2007-12-17 20:30:23 UTC
While we are on the sa-update issue, is there any reason that the cron file
/etc/cron.d/sa-upate is mode 600 and thus invisible to a non-root user? Wouldn't
644 be a bit more user friendly?

This together with the fact that the directory cron.d itself is mode 600 (I just
submitted a bugzilla on that) only made troubleshooting this problem all that
more difficult since I was not even familiar with the cron script since I do
actually try not to play around the system too much as roo.

Comment 17 Warren Togami 2007-12-17 21:53:04 UTC
> While we are on the sa-update issue, is there any reason that the cron file
> /etc/cron.d/sa-upate is mode 600 and thus invisible to a non-root user? Wouldn't
> 644 be a bit more user friendly?

It isn't even enabled by default.  It is commented out.  You need to be root to
uncomment it.  This is a non-issue.

Comment 18 Warren Togami 2007-12-17 22:14:55 UTC
Any idea how your affected systems became this way?  I've never seen this on any
of my systems, and I have never used sa-update --import before.

F8 x86_64 with SELinux enabled, sa-update works for me with no trouble. 

Comment 19 Need Real Name 2007-12-18 06:03:38 UTC
(In reply to comment #17)
> > While we are on the sa-update issue, is there any reason that the cron file
> > /etc/cron.d/sa-upate is mode 600 and thus invisible to a non-root user? Wouldn't
> > 644 be a bit more user friendly?
> 
> It isn't even enabled by default.  It is commented out.  You need to be root to
> uncomment it.  This is a non-issue.

I respectfully couldn't disagree more.
The whole reason for including options that are commented out is to allow the
knowledgeable reader to peruse the script and decide whether or not to enable th
functionality.

If the script is hidden from all non-root users then it is highly unlikely that
anyone will ever see the comment and have the chance to evaluate whether or not
to uncomment is. So why not just leave the file out in the first place and not
clutter up the system.
 
In fact, I am a case in point. I like the idea of updating spamassassin
automatically and am considering uncommenting this cron script. Since it is
hidden (and since I try to follow best practices of not running as root), I
would never have even learned about this option had this bug not come up!

Also, what is the big deal about changing the mode to 644 to be consistent with
other cron scripts especially since this bug report will likely require another
release of the rpm anyway?????

Comment 20 Need Real Name 2007-12-18 06:12:06 UTC
(In reply to comment #18)
> Any idea how your affected systems became this way?  I've never seen this on any
> of my systems, and I have never used sa-update --import before.
> 
> F8 x86_64 with SELinux enabled, sa-update works for me with no trouble. 

I'm now beginning to think that sa-update won't work on ANY system as configured
by the rpm unless EITHER the key is imported or the --nogpg flag is given.

If so, I would suggest that the line 
sa-update --import /usr/share/spamassassin/sa-update-pubkey.txt
be added to the postinstall script.

Luckily, repeated importation of the key doesn't change any of the content of
/etc/mail/spamassassin/sa-update-keys so you don't have to test each time
whether the key has been installed.

(note: I previously wrote sa-learn but meant to write sa-update)




Comment 21 Need Real Name 2007-12-18 06:19:21 UTC
(In reply to comment #18)
> Any idea how your affected systems became this way?  I've never seen this on any
> of my systems, and I have never used sa-update --import before.
> 
> F8 x86_64 with SELinux enabled, sa-update works for me with no trouble. 

Still working at figuring out how my system got that way.
To help me with my debugging, can you tell me whether the directory
/var/lib/spamassassin/3.002003 is ever even created on a fresh FC8 system (if
sa-update is not run)? I noticed that in FC6 at least, the directory
/var/lib/spamassassin is blank.


Comment 22 Need Real Name 2007-12-18 07:41:16 UTC
OK. I figured it out.
I opened up some old vmware images and found that sa-update was run first on Nov
12 -- only about 2 hours after booting F8 for the first time.

Interestingly, the initial installation seems to have pulled in a version of
spamassassin from ATrpms which indeed contains an UNCOMMENTED sa-update cron
file /etc/cron.daily/sa-update (note it was pulled in automatically since I had
added the ATrpms repo during the initial installation and the ATrpms version of
spamassassin must have been newer than the Fedora one)

Anyway, here is the ATrpms cron file:
#!/bin/bash
# Only restart spamd if sa-update returns 0, meaning it updated the rules
export HOME=/var/lib/spamassassin
cd $HOME
(/usr/bin/sa-update \
  --gpgkey D1C035168C1EBC08464946DA258CDB3ABDE9DC10 \
  --channel updates.spamassassin.org \
  --channel saupdates.openprotect.com \
  && /etc/init.d/spamassassin condrestart > /dev/null) 2>&1 \
  | tee -a /var/log/sa-update.log

(This explains why I now have the "saupdate.openprotect.com" related directory
and files in /var/lib/spamassassin/3.002003)

This was run by anacron about 2 hours after my initial boot and must have
crashed in an incomplete state.

I just re-created the initial state and verified that when the sa-update-keys
are reset to the original state and when /var/lib/spamassassin is reset to its
initial empty directory state, then the atrpms sa-update script does crash.

The reason is as follows. The gpg key explicitly presented in the ATrpms
cron.daily script is the gpg key for saupdates.openprotect.com so that part of
the sa-update goes fine. However, no gpg key is given (or present in
sa-update-keys) for the main updates.spamassassin.org ruleset.

As a result the sa-update utility only downloads the saupdates ruleset but
leaves out the base updates.spamassassin.org ruleset.

This creates a BAD situation since as mentioned above, once
/var/lib/spamassassin/3.002003 is created, it is used as the EXCLUSIVE source of
rules and overrides the other locations (e.g., /usr/share/spamassassin)

So all is now explained. The primary root cause does appear to be a bug in the
ATrpms version of spamassassin that presents a valid gpg key for only saupdates
but does not either explicitly present a key for updates.spamassassin or add it
to the keyring.

Still, this is a brittle situation even if one never uses ATrpms in that any
time the /var/lib/spamassassin/3.002003 directory is created it automatically
supersedes the /user/share/spamassassin ruleset.

One possible SOLUTION would be to preempt such problems by using the postinstall
scriptlets to create the /var/lib/spamassassin/3.002003 directory and then
create a link to the rules in /usr/share/spamassassin plus the packing list as
follows:
ln -s /usr/share/spamassassin updates_spamassassin_org
ls /usr/share/spamassassin | sed "s/^/include updates_spamassassin_org\//" >
updates_spamassassin_org.cf

Another perhaps better solution would be to just store the ruleset there in the
first place. Since otherwise anytime you update the rules in the spamassassin
rpm they will automatically be overriden by anybody who has EVER run sa-update
resulting in the creation of a non-empty 3.002003 directory.

For example, if I ran sa-update only once many months ago (and forgot about it
or didn't realize it) then no new spamassassin rpm rule updates will EVER be used.

So, basically, right now running sa-update can be very dangerous in that it may
cause the user to unwittingly create a fork between his upgrades and the rpm
upgrades -- all unknown to the non-expert user.

Comment 23 Warren Togami 2007-12-18 08:21:36 UTC
Excellent detailed analysis.

Note that I was correct as far back as Comment #4.  I suppose spamassassin
upstream can become more fault tolerant of this kind of error, but ultimately we
cannot control or support unpredictable behaviors caused by non-standard
component replacements from fork distributions.

http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5751
Upstream bug to make this more fault tolerant.

Comment 24 Need Real Name 2007-12-18 14:06:47 UTC
I agree that it should be more fault-tolerant. But there are also a couple of
Fedora-specific *installation* issues that could/should be fixed.

1. Install the gpg key so that sa-update actually works out-of-the-box. Since
you already are including the key in the rpm and since you have already set up
the keyrings why not help the user by installing the gpg key? All that is
required is inserting a one-liner into the postinstall script.

sa-learn --import /usr/share/spamassassin/sa-update-pubkey.txt

2. You still have the potentially significant issue that even if you use
sa-update just ONCE (and without any errors), spamassassin will SILENTLY IGNORE
ALL future rpm upgrades without EVER telling you. This situation will probably
not be addressed by the upstream bug a report. At a minimum, I would think that
the RPM install script should WARN of such a situation. Even better, have a test
in the postinstall script such as:

[ -e /var/lib/spamassassin/3.002003 ] && echo "WARNING: Running sa-update to
update rules in /var/lib/spamassassin/3.002003" && /usr/sbin/sa-update

This has the desirable side-effect that even if something does break during an
sa-update, it will at least be automagically fixed if/when spamassassin is
upgraded or reinstalled.

3. I still would argue strongly for changing the permissions on
/etc/cron.d/sa-update. The general philosophy of Fedora seems to be to keep all
system files readable unless there is a SPECIFIC vulnerability such as plaintext
passwords or other sensitive information. This cron script is about as generic
as they come. All that is required is to add the following to the %files

%attr(644,-,-) /etc/cron.d/sa-update

So... everything can be fixed by just adding 3 lines to the spec file :)

Comment 25 Warren Togami 2007-12-18 14:25:16 UTC
> 2. You still have the potentially significant issue that even if you use
> sa-update just ONCE (and without any errors), spamassassin will SILENTLY IGNORE
> ALL future rpm upgrades without EVER telling you.

Not true.  If you upgrade spamassassin to a new upstream version, it stops using
the old directory in /var/lib/spamassassin.

Fine, I will make it 644, but I don't have authorization to change the directory
permission.

Comment 26 Need Real Name 2007-12-18 18:37:41 UTC
Thanks.

For #2, I was talking about an rpm upgrade (not a spamassassin version upgrade).
Suppose for example that the RPM upgrade changes something that was broken
/usr/share/spamassassin, then the program spamassassin would have no way of
knowing about it.

Thanks for making change #3.
I still think that the proposed changes 1&2 are advantageous.
Taking into account your point about changes to Spamassassin version numbers, I
would slightly modify my proposed script for #2 so that the version number
(3.002003) is not hard-coded but is an rpm variable.


Comment 27 Bug Zapper 2008-11-26 08:37:18 UTC
This message is a reminder that Fedora 8 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 8.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '8'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 8's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 8 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 28 Kevin Fenzi 2008-12-16 00:42:29 UTC
ok, where are we here? Can this bug be closed now?

Comment 29 Bertil Askelid 2008-12-16 14:09:58 UTC
Bug has been fixed, no longer able to recreate it.