Suppose your system is set up to use Unix mailbox delivery (i.e., mailbox
files in /var/mail), and you use ipop3d (or perhaps imapd as well) to read
Now suppose someone sends you an E-mail message which has "Status: D" in
When ipop3d (and perhaps imapd as well -- I haven't tried this) reads your
mailbox, it will see the "Status: D", decide that the message has already
been deleted, and never show it to you.
This is not merely a hypothetical situation. I saw it happen to me today,
or at least I *think* that's what happened, although I can't prove that's
what happened in that particular case, since I lost the E-mail message. In
particular, I got E-mail from someone, I immediately fetched my new mail
using Emacs movemail's POP support, and yet there were no new messages in
my inbox. The log message in /var/log/maillog from when I fetched my mail
is particularly telling:
Aug 8 13:58:56 jik ipop3d: pop3 service init from 192.168.0.1
Aug 8 13:58:57 jik ipop3d: Login user=jik host=jik [192.168.0.1]
Aug 8 13:58:57 jik ipop3d: Logout user=jik host=jik [192.168.0.1]
If you look in the source code for ipop3d, you will see that the first
number after nmsgs is the number of available messages in the inbox, and
the second number is the number of messages that were found in the inbox,
some of which may be unavailable. As far as I can tell, the only time a
message can be in the inbox but unavailable is when it's deleted, and the
only time that can happen for a message you haven't actually looked at yet
is when "Status: D" is in its header when it arrives.
Once I guess that that's what happened, I was able to duplicate the
situation easily by sending myself a message with "Status: D" in its
Of course, other MTAs and MUAs should not be sending messages with "Status"
headers, because that's not a standard header. But there's nothing
stopping them from doing it, and certainly, an incoming message with
"Status: D" in its header shouldn't be dropped on the floor without ever
beeing seen by the recipient!
I've spent a long time today trying to figure out what the right solution
to this problem is. The best one that I've come up with is to modify all
the MTAs to strip the "status" field from incoming messages. I've done
this to sendmail, because that's the MTA I use, and I'll attach a patch.
Another idea I thought of is for ipop3d and other other imap daemons to
ignore the Status field if there isn't also an X-UID field. But, of
course, that doesn't really solve the problem, it just partially masks it,
because what if someone sends you an E-mail message which has both Status
and X-UID fields?
I suppose another possibility is for the imap package daemons to store in
the mailbox header the character position of the end of the mailbox as of
when they last saved it, and then to ignore Status, X-UID, etc. in any
message that comes after that character position when the mailbox is
I admit that this problem is somewhat obscure, but since it actually
happened to me today in a normal usage situation, I'd argue that it does
need to be fixed somehow.
Created attachment 26893 [details]
patch to make sendmail suppress "status" header
Any mail client could potentially send any mail header. If someone
sends a message with this header, what harm has it done?
I need to be really convinced that this is a problem. I don't see
it. I'm open though..
Yes, any mail client could send any header, but only a small subset of possible
headers are treated magically by the imap package. Since these headers are
magic to imap, something needs to be done to ensure that bogus values for them
are not fed into it.
It surprises me that you would argue that this is not a bug. It seems
manifestly obvious to me that it is unacceptable for mail accepted for delivery
by an SMTP server to never be seen by the user for which it was intended. Mail
must be either bounced or delivered; there is no other acceptable choice.
GIGO -> Garbage In, Garbage Out
Under what circumstances would this occur in reality?
1) Broken mail client sending mail with invalid header.
Solution: Fix the mail client
2) Someone doing it on purpose
For what gain?
What problem does it create?
What standard document does it violate?
I can't think of any. Don't send yourself mail with that header in it,
and if some app puts it there, it is broken - fix it. Still NOTABUG,
still open minded to be convinced if given sufficient evidence that
it causes a security threat or violates some standard (you have to
prove it to me that it is a problem, not me prove to you it isn't).
Are you sure you work for RedHat and not for Microsoft or IBM?
The attitude that it's OK for mail software to drop a message on the floor
without notifying anyone is endemic among developers of mail software for
Windows, but it is anathema to Unix software developers.
The mere fact that it is possible for someone to send me an E-mail message which
I will never see, even though they intended for me to see it, is sufficient
proof that this needs to be fixed. It is a problem because it can happen, and
what's more, it is a problem because it DID happen.
It's bogus to assert that a legitimate solution to this problem is to fix the
mail client, because (a) I don't have control over other people's mail clients
and (b) how can I know that it's happening in order to fix it if I never see the
mail messages in question?
As for why someone would do it on purpose, well, I can think of two reasons off
the top of my head:
1) Let's say someone discovers a buffer overrun bug in one of the mail daemons.
To exploit it, all one needs to do is get the daemon to parse a mail message.
So, you send someone a mesage which has a header in it which will take advantage
of this bug, but you ALSO put "Status: D" in the header of the message. When
they fetch their mail, the security hole will be tickled when the message is
parsed, but they'll never see the message to know that it happened.
2) Suppose you want to get someone in trouble. Suppose you want to accuse them
of exchanging child pornography with other people using a company E-mail
account. So you send them a piece of child pornography by E-mail, suitably
redirected to hide your identity, and put "Status: D" in it. Then you send an
anonymous top to their sysadmins to check out the porn in their mailbox. Now,
if the admin gets to it first, they'll see the porn and all hell will break
loose. But if the user fetches their mail first, they'll never see it.
As for why someone would do it by accident, they can do that merely by using
rmail-resend from inside Emacs after fetching their mail with movemail.
It's worse than I thought. The same problem occurs if the incoming E-mail has
"x-status" as well as "status".
While it's true that putting "status" in an E-mail message isn't really OK
because headers not specified by the RFCs are supposed to start with "x-",
putting "x-status" in an E-mail message *is* really OK, because it's legal for
any mail client to insert any "x-*" header. Therefore, this bug causes imap to
discard messages which are COMPLETELY LEGAL.
I will attach an updated sendmail patch which filters out x-status in addition
Created attachment 27053 [details]
patch to suppress x-status as well as status
I still don't consider this a bug or flaw at all.. One think I know
for sure, is that sendmail patches do not apply to imap source code.
I do not maintain sendmail. So, this is not my decision, it is the
decision of Florian or someone else. I am reassigning the bug to
the sendmail component, and leaving this up to him.
One thing I still have yet to hear, is what problem this causes and
why it is bad. ie: if someone did do this on purpose, what benefit
is there to them, and what harm is there to the recipient. I see none.
Anyway, I now digress... ;0)
This is not a sendmail bug. This is an imap bug. I supplied the sendmail patch
merely as a workaround for the imap bug.
One could argue that the sendmail patch I supplied introduces a bug into
sendmail!, since it's not legitimate for sendmail to strip "x-status" headers
when they're perfectly valid RFC 822 headers. The real fix is not to modify
sendmail but rather to modify imap not to store status information in a way that
can be spoofed.
+One thing I still have yet to hear, is what problem this causes and
+why it is bad. ie: if someone did do this on purpose, what benefit
+is there to them, and what harm is there to the recipient. I see none.
+Anyway, I now digress... ;0)
I do not understand how you say what you said in the paragraph above.
What problem does it cause? Lost mail. Why is this bad? Because it is not
acceptable to ever lose mail without notifying somebody. Why would someone do
this on purpose? I offered two examples of why, off the top of my head, in a
comment I put in the bug yesterday. But that's not even the point, because it's
also completely feasible to do it by *accident* and cause mail to be lost.
I strongly disagree with your assertion that this is not a bug and your
assertion that it is not a bug in imap. I do not know if there is a mechanism
for "appealing" the decisions of individual developers at RedHat, but if there
is, I would politely like to request that your decision not to pursue this as an
imap bug be escalated to your management, because I think it's wrong.
I reassigned it to sendmail, because I didn't see an imap bug,
and figured that since you attached sendmail patch, you thought
that was the proper fix, which should have been filed against
sendmail. Rather than closing the bug, I reassigned it for you.
Now I'm reassigning it back because you do not believe the
sendmail hack is the correct fix anyway.
Yes, you offered two examples of why. No I don't see why either makes
any difference. Please realize jik, that I _have_not_ *CLOSED* this
bug report as NOTABUG or WONTFIX. Why? Because I am open minded.
There is a chance my viewpoint may be wrong or I would have just said
NOTABUG and not commented on it again period. Instead, I have give
my viewpoint, and left it open pending discussion about it both internally
and with others in the community. Whatever decision that is made,
will be made sensibly. If Red Hat collectively as a group along with
the community see a bug that needs fixing WRT this, then it will most
likely happen. If we do not, then it wont.
No need to pull "I'm going for your manager then" games.. I appreciate
your bug reports and contributions, but please appreciate that every bug
reported by someone is not necessarily a bug. If one perceives something
to not be a bug, then they need to be provided with for example: exploit
code and/or in depth realistic "this is really bad because" explanation.
As I said, this is being investigated currently. The result of this
investigation will decide the final resolution of the report.
This has been reported upstream to UW, and discussed in various forums.
If UW updates this upstream, it will be carried into Red Hat Linux when
the newer version of imap containing the change gets updated. Since this
issue touches upon topics of complex disagreement with various parties,
I feel it is an issue the upstream developers should decide upon. As such,
there wont be any local Red Hat changes made without approval of the
University of Washington development team. I'm closing the bug, but again,
if the upstream source changes, we'll pick up the change.