Bug 392851
Summary: | spamassassin doesn't detect its own sample-spam as spam | ||||||
---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Bertil Askelid <bertil> | ||||
Component: | spamassassin | Assignee: | Warren Togami <wtogami> | ||||
Status: | CLOSED NOTABUG | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
Severity: | urgent | Docs Contact: | |||||
Priority: | low | ||||||
Version: | 8 | CC: | bugzilla | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | i386 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2008-12-16 14:09:58 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Bertil Askelid
2007-11-20 19:12:51 UTC
> spamc -f < /usr/share/doc/spamassassin-3.2.3/sample-spam.txt
What is -f supposed to do? I don't see -f in spamc's man page.
Furthermore, trying this on my system works just fine both with and without -f.
You are correct. -f is not needed. I ran spamc without -f on both my FC8 and my FC7 machines. Same result as before. Fails to detect it as a spam on FC8, while FC7 is just fine. I've just installed FC8 from the DVD and then done yum from Fedora, Livnia, ATrpms, and Freshrpms to update my system with the latest. As it works for you, there must be something that collides with spamassassin on my system. Please advise me on how to debug spamassassin to figure out and correct. Thanks! Downgraded to the spamassassin-3.2.0-1.fc7 from the FC7 release DVD. Works just fine on my FC8 system. ~> spamc < /usr/share/doc/spamassassin-3.2.0/sample-spam.txt X-Spam-Flag: YES X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on z.lan X-Spam-Level: ************************************************** X-Spam-Status: Yes, score=999.9 required=5.0 tests=BAYES_00,GTUBE,NO_RECEIVED, NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,RAZOR2_CHECK autolearn=no version=3.2.0 X-Spam-Report: * -0.0 NO_RELAYS Informational: message was not relayed via SMTP * 1000 GTUBE BODY: Generic Test for Unsolicited Bulk Email * -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1% * [score: 0.0000] * 1.5 RAZOR2_CF_RANGE_E4_51_100 Razor2 gives engine 4 confidence level * above 50% * [cf: 100] * 0.5 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/) * 0.5 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50% * [cf: 100] * -0.0 NO_RECEIVED Informational: message has no Received headers Subject: [SPAM] Test spam mail (GTUBE) Message-ID: <GTUBE1.1010101> Date: Wed, 23 Jul 2003 23:30:00 +0200 From: Sender <sender> To: Recipient <recipient> Precedence: junk MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Prev-Subject: Test spam mail (GTUBE) This is the GTUBE, the Generic Test for Unsolicited Bulk Email If your spam filter supports it, the GTUBE provides a test by which you can verify that the filter is installed correctly and is detecting incoming spam. You can send yourself a test mail containing the following string of characters (in upper case and with no white spaces and line breaks): XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X You should send this test mail from an account outside of your network. You mention AtRPMS. AtRPMS is effectively a fork of Fedora because it replaces system packages. Please see if you can reproduce this bug with stock Fedora 8 packages. Nobody else is able to reproduce it with stock Fedora 8. If an AtRPMS package is causing this failure, please report the bug to that project. > You mention AtRPMS. AtRPMS is effectively a fork of Fedora because it
> replaces system packages.
Off topic to the actual bug report, but that's not really a sane conclusion.
Even fedora.us from where Waren stems from replaced system packages even more
hard core packages like shadow-utils. Was fedora.us a fork of Red Hat Linux? And
every other repo out there had to do the same in similar situation including
dag, freshrpms, livna etc. replacing rpm itself or mp3-castrated software. So
the first one not having had to replace a RHL or Fedora package should throw the
first stone.
ATrpms is a generic 3rd party add-on repo improving and enhancing your Fedora
and/or RHEL experience. If ATrpms ever wants to fork something you'll know.
Back on-topic: Bertil you need to make sure you actually have the spamassassin
service running. Like "service spamassassin start". Since you have the same
issue with Fedora 8's plain and ATrpms' enhanced spamassassin packages, but not
FC7's, it looks like either a non-started service or a misconfigration
somewhere. It would be strange if the F8 plain spamassassin package would be
confused by presence of other helper software and F7's not (but nothing can be
outruled).
Since you filed a bug over at ATrpms we can follow up there in case you find out
that there really is an issue with ATrpms' packages after having restarted the
spamassassin service (spamd). But I wouldn't be getting any mails if the
spamassassin & support packages at ATrpms were that broken. ;)
> Off topic to the actual bug report, but that's not really a sane conclusion.
> Even fedora.us from where Waren stems from replaced system packages even more
> hard core packages like shadow-utils.
Give me a break. shadow-utils in fedora.us contained a one-liner patch to fix a
segfault crasher. fedora.us replaced so few things, and when it did it did in
some insignificant way.
Steps taken to investigate bug. This shows that the spamd server indeed is running while the bug happens. As you can see, the FC8 versions of spamassassin from ATrpms (spamassassin-3.2.3-44.fc8.i386.rpm) and Fedora (spamassassin-3.2.3-2.fc8.i386.rpm) both fail while the one from FC7 (spamassassin-3.2.0-1.fc7.i386.rpm) works just fine. bertil| rpm -e spamassassin bertil| rpm -ivh spamassassin-3.2.3-44.fc8.i386.rpm Preparing... ########################################### [100%] 1:spamassassin ########################################### [100%] bertil| /etc/rc.d/init.d/spamassassin start Starting spamd: [ OK ] bertil| spamc < /usr/share/doc/spamassassin-*/sample-spam.txt X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on z.lan Subject: Test spam mail (GTUBE) Message-ID: <GTUBE1.1010101> Date: Wed, 23 Jul 2003 23:30:00 +0200 From: Sender <sender> To: Recipient <recipient> Precedence: junk MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit This is the GTUBE, the Generic Test for Unsolicited Bulk Email If your spam filter supports it, the GTUBE provides a test by which you can verify that the filter is installed correctly and is detecting incoming spam. You can send yourself a test mail containing the following string of characters (in upper case and with no white spaces and line breaks): XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X You should send this test mail from an account outside of your network. bertil| rpm -e spamassassin bertil| rpm -ivh spamassassin-3.2.3-2.fc8.i386.rpm Preparing... ########################################### [100%] 1:spamassassin ########################################### [100%] bertil| /etc/rc.d/init.d/spamassassin start Starting spamd: [ OK ] bertil| spamc < /usr/share/doc/spamassassin-*/sample-spam.txt X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on z.lan Subject: Test spam mail (GTUBE) Message-ID: <GTUBE1.1010101> Date: Wed, 23 Jul 2003 23:30:00 +0200 From: Sender <sender> To: Recipient <recipient> Precedence: junk MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit This is the GTUBE, the Generic Test for Unsolicited Bulk Email If your spam filter supports it, the GTUBE provides a test by which you can verify that the filter is installed correctly and is detecting incoming spam. You can send yourself a test mail containing the following string of characters (in upper case and with no white spaces and line breaks): XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X You should send this test mail from an account outside of your network. bertil| rpm -e spamassassin bertil| rpm -ivh spamassassin-3.2.0-1.fc7.i386.rpm Preparing... ########################################### [100%] 1:spamassassin ########################################### [100%] bertil| /etc/rc.d/init.d/spamassassin start Starting spamd: [ OK ] bertil| spamc < /usr/share/doc/spamassassin-*/sample-spam.txt X-Spam-Flag: YES X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on z.lan X-Spam-Level: ************************************************** X-Spam-Status: Yes, score=1002.5 required=5.0 tests=GTUBE,NO_RECEIVED, NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,RAZOR2_CHECK autolearn=no version=3.2.0 X-Spam-Report: * -0.0 NO_RELAYS Informational: message was not relayed via SMTP * 1000 GTUBE BODY: Generic Test for Unsolicited Bulk Email * 1.5 RAZOR2_CF_RANGE_E4_51_100 Razor2 gives engine 4 confidence level * above 50% * [cf: 100] * 0.5 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/) * 0.5 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50% * [cf: 100] * -0.0 NO_RECEIVED Informational: message has no Received headers Subject: Test spam mail (GTUBE) Message-ID: <GTUBE1.1010101> Date: Wed, 23 Jul 2003 23:30:00 +0200 From: Sender <sender> To: Recipient <recipient> Precedence: junk MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit This is the GTUBE, the Generic Test for Unsolicited Bulk Email If your spam filter supports it, the GTUBE provides a test by which you can verify that the filter is installed correctly and is detecting incoming spam. You can send yourself a test mail containing the following string of characters (in upper case and with no white spaces and line breaks): XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X You should send this test mail from an account outside of your network. bertil| Bertil, in order to prove that ATrpms has nothing to do with this bug, please just remove all mail related ATrpms packages and show that this still happens, or just remove all ATrpms packages just to make it clear that the bug is in one or several pure Fedora shipped packages w/o any 3rd party interaction. At least I can't reproduce it on F8 i386/x86_64/ppc as well as several other distros I'm using the same packaging. If you continue posting examples about using ATrpms you give a convenient excuse for non-ATrpms fans to look away. I have removed *all* ATrpms and now been running my spam test as described above on a *clean* Fedora Core 8 system. Same result as reported before. Only spamassassin-3.2.0-1.fc7.i386.rpm detects the spam example as spam. Created attachment 286651 [details]
spamassassin -D output
I am having the same problem - spamassassin is COMPLETELY BROKEN on F8 -- all mail passes through with zero spam score vs. my previous FC6 installation where it caught almost all spam. IMHO, this blaming stuff on ATrpms is a red herring. There is DEFINITELY something majorly screwed with spamassasin on F8 (or a conflict with some other setup configuration). Basically all mail gives a zero score. I have attached the debug output from running on /usr/share/doc/spamassassin-3.2.3/sample-spam.txt (but unfortunately due to a type it precedes this comment) Any suggestions on how to debug this? OK I THINK I SOLVED THIS (after many wasted hours). The following thread gave me the answer. http://mail-archives.apache.org/mod_mbox/spamassassin-users/200605.mbox/%3c6bb609560605131018i2a51ff1ex42fe83fdee0eb4f8@mail.gmail.com%3e Apparently spamassassin looks successively in the following directories for the first non-empty directory that if present will override configuration data: =item /var/lib/spamassassin/3.002003 =item /usr/share/spamassassin =item /usr/share/spamassassin =item /usr/local/share/spamassassin =item /usr/share/spamassassin (note /usr/share/spamassassin is repeated 3 times) Well, I happen to have /var/lib/spamassassin/3.002003 on my new F8 system (but no equivalent on my old F6 system). This directory is populated by sa-update -- but apparently there is a bug that sometimes the directory still is created even if sa-update fails to finish downloading an update. Now, sa-update is run daily at 4:10 using a crontab entry "hidden" in the read-protected directory /etc/cron.d/ (not sure why it doesn't use cron.daily like everything else). Looking in my /var/log/sa-update.log file, I noticed the following error which presumably is responsible for the faulty updating (and the incomplete creation of the /var/lib/spamassassin/3.002003 directory) that causes spamassassin to choke: error: GPG validation failed! The update downloaded successfully, but it was not signed with a trusted GPG key. Instead, it was signed with the following keys: 24F434CE Perhaps you need to import the channel's GPG key? For example: wget http://spamassassin.apache.org/updates/GPG.KEY sa-update --import GPG.KEY channel: GPG validation failed, channel failed Now I'm not sure why I don't have the right GPG key. (It's also not clear to me how this GPG key gets imported for anybody since I don't see it being done in the rpm install scripts or being part of the rpm itself) In any case, I was able to fix it by running: sa-learn --import /usr/share/spamassassin/sa-update-pubkey.txt Interestingly, I'm not sure why the following doesn't work (giving the same GPG error): sa-update --gpgpkey /usr/share/spamassassin/sa-update-pubkey.txt Note though that running 'sa-update --nogpg' did work. So in summary, I conclude the following: 1. At least on my system, the GPG key was not imported properly. Not sure why and not sure how many other people this affects 2. My problem where spamassassin broke completely was due to a combination of an incomplete sa-update (which caused the creation of a partial 3.002003 directory) plus the lack of a gpg key which prevented subsequent good sa-updates from overwriting the complete one. Hope this helps... This *did* really help. Problem solved. Latest FC8 spamassassin works just fine. So the install procedure in the spamassassin.rpm need to corrected to handle the key setup. Thanks! Warren, could you open an upstream bug about this? sa-update should not leave the system in a state where there are no valid rules files, regardless of whether the key was imported or not. While we are on the sa-update issue, is there any reason that the cron file /etc/cron.d/sa-upate is mode 600 and thus invisible to a non-root user? Wouldn't 644 be a bit more user friendly? This together with the fact that the directory cron.d itself is mode 600 (I just submitted a bugzilla on that) only made troubleshooting this problem all that more difficult since I was not even familiar with the cron script since I do actually try not to play around the system too much as roo. > While we are on the sa-update issue, is there any reason that the cron file
> /etc/cron.d/sa-upate is mode 600 and thus invisible to a non-root user? Wouldn't
> 644 be a bit more user friendly?
It isn't even enabled by default. It is commented out. You need to be root to
uncomment it. This is a non-issue.
Any idea how your affected systems became this way? I've never seen this on any of my systems, and I have never used sa-update --import before. F8 x86_64 with SELinux enabled, sa-update works for me with no trouble. (In reply to comment #17) > > While we are on the sa-update issue, is there any reason that the cron file > > /etc/cron.d/sa-upate is mode 600 and thus invisible to a non-root user? Wouldn't > > 644 be a bit more user friendly? > > It isn't even enabled by default. It is commented out. You need to be root to > uncomment it. This is a non-issue. I respectfully couldn't disagree more. The whole reason for including options that are commented out is to allow the knowledgeable reader to peruse the script and decide whether or not to enable th functionality. If the script is hidden from all non-root users then it is highly unlikely that anyone will ever see the comment and have the chance to evaluate whether or not to uncomment is. So why not just leave the file out in the first place and not clutter up the system. In fact, I am a case in point. I like the idea of updating spamassassin automatically and am considering uncommenting this cron script. Since it is hidden (and since I try to follow best practices of not running as root), I would never have even learned about this option had this bug not come up! Also, what is the big deal about changing the mode to 644 to be consistent with other cron scripts especially since this bug report will likely require another release of the rpm anyway????? (In reply to comment #18) > Any idea how your affected systems became this way? I've never seen this on any > of my systems, and I have never used sa-update --import before. > > F8 x86_64 with SELinux enabled, sa-update works for me with no trouble. I'm now beginning to think that sa-update won't work on ANY system as configured by the rpm unless EITHER the key is imported or the --nogpg flag is given. If so, I would suggest that the line sa-update --import /usr/share/spamassassin/sa-update-pubkey.txt be added to the postinstall script. Luckily, repeated importation of the key doesn't change any of the content of /etc/mail/spamassassin/sa-update-keys so you don't have to test each time whether the key has been installed. (note: I previously wrote sa-learn but meant to write sa-update) (In reply to comment #18) > Any idea how your affected systems became this way? I've never seen this on any > of my systems, and I have never used sa-update --import before. > > F8 x86_64 with SELinux enabled, sa-update works for me with no trouble. Still working at figuring out how my system got that way. To help me with my debugging, can you tell me whether the directory /var/lib/spamassassin/3.002003 is ever even created on a fresh FC8 system (if sa-update is not run)? I noticed that in FC6 at least, the directory /var/lib/spamassassin is blank. OK. I figured it out. I opened up some old vmware images and found that sa-update was run first on Nov 12 -- only about 2 hours after booting F8 for the first time. Interestingly, the initial installation seems to have pulled in a version of spamassassin from ATrpms which indeed contains an UNCOMMENTED sa-update cron file /etc/cron.daily/sa-update (note it was pulled in automatically since I had added the ATrpms repo during the initial installation and the ATrpms version of spamassassin must have been newer than the Fedora one) Anyway, here is the ATrpms cron file: #!/bin/bash # Only restart spamd if sa-update returns 0, meaning it updated the rules export HOME=/var/lib/spamassassin cd $HOME (/usr/bin/sa-update \ --gpgkey D1C035168C1EBC08464946DA258CDB3ABDE9DC10 \ --channel updates.spamassassin.org \ --channel saupdates.openprotect.com \ && /etc/init.d/spamassassin condrestart > /dev/null) 2>&1 \ | tee -a /var/log/sa-update.log (This explains why I now have the "saupdate.openprotect.com" related directory and files in /var/lib/spamassassin/3.002003) This was run by anacron about 2 hours after my initial boot and must have crashed in an incomplete state. I just re-created the initial state and verified that when the sa-update-keys are reset to the original state and when /var/lib/spamassassin is reset to its initial empty directory state, then the atrpms sa-update script does crash. The reason is as follows. The gpg key explicitly presented in the ATrpms cron.daily script is the gpg key for saupdates.openprotect.com so that part of the sa-update goes fine. However, no gpg key is given (or present in sa-update-keys) for the main updates.spamassassin.org ruleset. As a result the sa-update utility only downloads the saupdates ruleset but leaves out the base updates.spamassassin.org ruleset. This creates a BAD situation since as mentioned above, once /var/lib/spamassassin/3.002003 is created, it is used as the EXCLUSIVE source of rules and overrides the other locations (e.g., /usr/share/spamassassin) So all is now explained. The primary root cause does appear to be a bug in the ATrpms version of spamassassin that presents a valid gpg key for only saupdates but does not either explicitly present a key for updates.spamassassin or add it to the keyring. Still, this is a brittle situation even if one never uses ATrpms in that any time the /var/lib/spamassassin/3.002003 directory is created it automatically supersedes the /user/share/spamassassin ruleset. One possible SOLUTION would be to preempt such problems by using the postinstall scriptlets to create the /var/lib/spamassassin/3.002003 directory and then create a link to the rules in /usr/share/spamassassin plus the packing list as follows: ln -s /usr/share/spamassassin updates_spamassassin_org ls /usr/share/spamassassin | sed "s/^/include updates_spamassassin_org\//" > updates_spamassassin_org.cf Another perhaps better solution would be to just store the ruleset there in the first place. Since otherwise anytime you update the rules in the spamassassin rpm they will automatically be overriden by anybody who has EVER run sa-update resulting in the creation of a non-empty 3.002003 directory. For example, if I ran sa-update only once many months ago (and forgot about it or didn't realize it) then no new spamassassin rpm rule updates will EVER be used. So, basically, right now running sa-update can be very dangerous in that it may cause the user to unwittingly create a fork between his upgrades and the rpm upgrades -- all unknown to the non-expert user. Excellent detailed analysis. Note that I was correct as far back as Comment #4. I suppose spamassassin upstream can become more fault tolerant of this kind of error, but ultimately we cannot control or support unpredictable behaviors caused by non-standard component replacements from fork distributions. http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5751 Upstream bug to make this more fault tolerant. I agree that it should be more fault-tolerant. But there are also a couple of Fedora-specific *installation* issues that could/should be fixed. 1. Install the gpg key so that sa-update actually works out-of-the-box. Since you already are including the key in the rpm and since you have already set up the keyrings why not help the user by installing the gpg key? All that is required is inserting a one-liner into the postinstall script. sa-learn --import /usr/share/spamassassin/sa-update-pubkey.txt 2. You still have the potentially significant issue that even if you use sa-update just ONCE (and without any errors), spamassassin will SILENTLY IGNORE ALL future rpm upgrades without EVER telling you. This situation will probably not be addressed by the upstream bug a report. At a minimum, I would think that the RPM install script should WARN of such a situation. Even better, have a test in the postinstall script such as: [ -e /var/lib/spamassassin/3.002003 ] && echo "WARNING: Running sa-update to update rules in /var/lib/spamassassin/3.002003" && /usr/sbin/sa-update This has the desirable side-effect that even if something does break during an sa-update, it will at least be automagically fixed if/when spamassassin is upgraded or reinstalled. 3. I still would argue strongly for changing the permissions on /etc/cron.d/sa-update. The general philosophy of Fedora seems to be to keep all system files readable unless there is a SPECIFIC vulnerability such as plaintext passwords or other sensitive information. This cron script is about as generic as they come. All that is required is to add the following to the %files %attr(644,-,-) /etc/cron.d/sa-update So... everything can be fixed by just adding 3 lines to the spec file :) > 2. You still have the potentially significant issue that even if you use
> sa-update just ONCE (and without any errors), spamassassin will SILENTLY IGNORE
> ALL future rpm upgrades without EVER telling you.
Not true. If you upgrade spamassassin to a new upstream version, it stops using
the old directory in /var/lib/spamassassin.
Fine, I will make it 644, but I don't have authorization to change the directory
permission.
Thanks. For #2, I was talking about an rpm upgrade (not a spamassassin version upgrade). Suppose for example that the RPM upgrade changes something that was broken /usr/share/spamassassin, then the program spamassassin would have no way of knowing about it. Thanks for making change #3. I still think that the proposed changes 1&2 are advantageous. Taking into account your point about changes to Spamassassin version numbers, I would slightly modify my proposed script for #2 so that the version number (3.002003) is not hard-coded but is an rpm variable. This message is a reminder that Fedora 8 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 8. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '8'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 8's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 8 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping ok, where are we here? Can this bug be closed now? Bug has been fixed, no longer able to recreate it. |