Bug 1305061

(In reply to Sjoerd Mullender from comment #0)
> Description of problem:
> I don't know whether this bug is in bugzilla or perl-Encode, but since the
> update to perl-Encode-2.80-4.fc23.x86_64 my bugzilla installation produces
> emails that are horrible.  I suspect the this commit:
> https://github.com/dankogai/p5-encode/commit/
> 77c0a92d8d12f06a3d92ea3d798ec81170b9202b
> Mails sent by bugzilla contain a Date header that my thunderbird can't
> parse, the To address is mangled, and all headers that bugzilla creates use
> quoted-printable encoding.
> An example:
> To: =?UTF-8?Q?bugs=2Dsql=40monetdb=2Eorg?=@monetdb.org
> Subject:
> =?UTF-
> 8?Q?=5BBug=203916=5D=20New=3A=20consideration=20for=20MonetDBLite=3A=20manage
> =20database=20updates=20properly?=
> Date: =?UTF-8?Q?Thu=2C=2004=20Feb=202016=2011=3A02=3A14=20=2B0000?=
> X-Bugzilla-Reason: =?UTF-8?Q?AssignedTo?=
> 
Could you please describe what the data looks before encoding? Without knowing the input, I cannot know what the correct output should be expected.

> All headers that bugzilla produces (as opposed to the ones the MTA adds)
> use quoted-printable encoding, even when they are fully ASCII.

This has been reported to upstream as <https://rt.cpan.org/Public/Bug/Display.html?id=111853>. It looks ugly, but it does not violate any specification. If Bugzilla does not want to encode the e-mail addresses, then it should not submit them to MIME-encoding function.

(In reply to Petr Pisar from comment #1)
> (In reply to Sjoerd Mullender from comment #0)
> > Description of problem:
> > I don't know whether this bug is in bugzilla or perl-Encode, but since the
> > update to perl-Encode-2.80-4.fc23.x86_64 my bugzilla installation produces
> > emails that are horrible.  I suspect the this commit:
> > https://github.com/dankogai/p5-encode/commit/
> > 77c0a92d8d12f06a3d92ea3d798ec81170b9202b
> > Mails sent by bugzilla contain a Date header that my thunderbird can't
> > parse, the To address is mangled, and all headers that bugzilla creates use
> > quoted-printable encoding.
> > An example:
> > To: =?UTF-8?Q?bugs=2Dsql=40monetdb=2Eorg?=@monetdb.org
> > Subject:
> > =?UTF-
> > 8?Q?=5BBug=203916=5D=20New=3A=20consideration=20for=20MonetDBLite=3A=20manage
> > =20database=20updates=20properly?=
> > Date: =?UTF-8?Q?Thu=2C=2004=20Feb=202016=2011=3A02=3A14=20=2B0000?=
> > X-Bugzilla-Reason: =?UTF-8?Q?AssignedTo?=
> > 
> Could you please describe what the data looks before encoding? Without
> knowing the input, I cannot know what the correct output should be expected.

To: bugs-sql@monetdb.org
Subject: [Bug 3916] New: consideration for MonetDBLite: manage database updates properly
Date: Thu, 04 Feb 2016 11:02:14 +0000
X-Bugzilla-Reason: AssignedTo

> > All headers that bugzilla produces (as opposed to the ones the MTA adds)
> > use quoted-printable encoding, even when they are fully ASCII.
> 
> This has been reported to upstream as
> <https://rt.cpan.org/Public/Bug/Display.html?id=111853>. It looks ugly, but
> it does not violate any specification. If Bugzilla does not want to encode
> the e-mail addresses, then it should not submit them to MIME-encoding
> function.

That why I wrote that I didn't know whether this bug is in Bugzilla or perl-Encode.  ;-)

I don't know what the API is, but it seems wrong if the Date and To fields get mangled the way they were.  The To field given to perl-Encode was the address bugs-sql, this was quoted by perl-Encode to =?UTF-8?Q?bugs=2Dsql=40monetdb=2Eorg?= which was then not recognized by the MTA as a fully qulified email address, so it added a domain, hence the two @ signs in the deocded result.

Upstream completely rewrote the MIME coding in Encode-2.83. The old code was one big mess with corner cases violating RFC 2047. I will check if it could help you.

2.83 still encodes everything.

Created attachment 1144849 [details]
possible patch to fix the problem

Here is a patch that I hope fixes the problem.  I haven't tried it out for real, but the reported failure when I run "make test" looks like it does the right thing.

What the code did before I intervened is to take the whole header value (i.e. everything after the first colon) and encode it in its entirety as UTF-8 quoted printable.  What I did instead is to take that complete value, split it on white space, and for each chunk check whether it is pure ASCII (range ! to ~) and if so, pass it unchanged.  Otherwise use the old code to encode the piece.  Then at the end, join all the pieces (encoded and unchanged) with a space between them.

A proper Perl programmer might want to take a look whether this is any good.

The code is based on the patched source from perl-Encode-2.80-4.fc23.src.rpm (i.e., I ran rpmbuild -bp).

Splitting on spaces and then concatenating is problematic. ASCI SPACE QUOTED sequence and QUOTED SPACE QUOTED sequence has different rules for ignoring the SPACE in between.

The Encode::MIME::Header was broken because of this and the 2.83 fixed it finally. The resolution is that Encode::MIME::Header will always encode whole header value as one chunk. If an application wants to encode headers with e-mail addresses (e.g. To:), it must do the segmentation by itself and pass for encoding only the interested parts. See <https://rt.cpan.org/Public/Bug/Display.html?id=111853> and a fix for a broken Encode::MIME::Header user <https://github.com/theory/svn-notify/pull/15>.

I don't know yet what to do in Fedora 23. Maybe I will applyr you patch to restore older behavior in the expense that the encoding will be broken in some cases.

But Fedora >= 24 will include the 2.83 version that does not do the segmentation and the one who must be fixed there will be the applications.

I'm sorry upgrading the perl-Encode in Fedora 23 caused problems to you, I will try to restore the old behavior thankfully to your patch.

Please talk to Encode developers in <https://rt.cpan.org/Public/Bug/Display.html?id=111853>. They are experts in this area and I believe they can explain the Encode design better than me and can help you with developing the best fix for you applications.

As I wrote in the original bug report, I didn't know whether the problem was in perl-Encode or bugzilla.  It's the combination of the two that causes the headaches.  If this is not actually a bug in perl-Encode, then this is a bug in bugzilla.  It should do the splitting before handing the headers over to perl-Encode.

I tried your patch. All the places where Encode-2.80 tests fail are expected except this one:

To: dankogai.jp (小飼=Kogai, 弾=Dan)

The original 2.80 code produces:

To: =?UTF-8?Q?dankogai=40dan=2Eco=2Ejp=20=28?=
 =?UTF-8?Q?=E5=B0=8F=E9=A3=BC=3DKogai=2C=20=E5=BC=BE=3DDan?= =?UTF-8?Q?=29?=

The patched 2.80 code produces:

To: dankogai.jp (
 =?UTF-8?Q?=E5=B0=8F=E9=A3=BC=3DKogai=2C?= =?UTF-8?Q?=E5=BC=BE=3DDan?=)

That means it inserts a space between "(" and "小". This is because splitting the text into multiple lines happens before calling _encode_q(). The line splitting code expects all the lines will start and end with encoded-word (=?...?=) so the line break will be ignored when decoding.

But in this case you have a plain line followed by an encoded line, and in this case the line break is decoded as a space.

I feel it's not possible to fix Encode-2.80 to restore original behavior without introducing bugs.

If your Bugzilla installation comes from Fedora's package, please file a bug against bugzilla component to fix how Bugzilla uses Encode::encode('MIME-Q').

I don't agree with the resolution.  It *is* a bug.  Maybe not in perl-Encode, but then in bugzilla.  So this bug report should then be assigned to the bugzilla component.

I can confirm that this is a bug with F23 and the latest shipped Bugzilla release 4.4.11-1.fc23. I upgraded to bugzilla-5.0.2-2.fc24 and the issue persists.

A downgrade to perl-Encode-2.78-2.fc23 fixes the problem.

I would like to ask you to re-open this bug and assign to someone who is either familiar with perl-Encode and/or Bugzilla.

Reopening and assigning this to bugzilla.

perl-Encode 2.79 is the last version that worked with Bugzilla. Version 2.80 and up to 2.84 are broken.

From the #bugzilla IRC channel:

21:08 <@dylan> eseyman: yes, that is a known bug in that module
21:08 <@dylan> we will have to work around it, the best solution is downgrading it at the moment

(In reply to Emmanuel Seyman from comment #13)
> 21:08 <@dylan> eseyman: yes, that is a known bug in that module
> 21:08 <@dylan> we will have to work around it, the best solution is
> downgrading it at the moment

This is not a bug in perl-Encode. It was a bug in the Bugzilla code itself which has been fixed a month ago, see:

https://bugzilla.mozilla.org/show_bug.cgi?id=1246228
https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=commitdiff;h=652ee918

This fix will be in Bugzilla 5.0.3.

bugzilla-5.0.2-3.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3527028bbe

Is there also going to come a fix for Fedora 23?

I've applied the patch in rawhide and issued an update for f24.
@lpsolit, is there a reason Bugzilla's 4.4 branch didn't receive the fix?

bugzilla-5.0.2-3.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-3527028bbe

bugzilla-5.0.2-3.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

It's nice that the bug is fixed on Fedora 24, but I'm running Fedroa 23, and I reported the bug for Fedora 23.  So I have to reopen this.

bugzilla-4.4.11-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-bcffbf3c08

(In reply to Sjoerd Mullender from comment #20)
>
> It's nice that the bug is fixed on Fedora 24, but I'm running Fedroa 23, and
> I reported the bug for Fedora 23.  So I have to reopen this.

Indeed. I thought putting the bug back to the ASSIGNED status would prevent bodhi from touching it again but it looks like I was wrong.

Okay, given the lack of activity of bmo's bug #1246228, I've decided to include Frédéric's submitted patch. Sjoerd, testing the update I just put out would be very appreciated (it works for me but it's more important that it works for you).

Testing will have to wait until (well) after the weekend.  Monday is a holiday here, and we're making use of it.

Thank you for fixing the Bugzilla.

bugzilla-4.4.11-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-bcffbf3c08

bugzilla-4.4.12-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6cdcddef2c

bugzilla-4.4.12-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6cdcddef2c

bugzilla-4.4.12-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.