Bug 1305061 - perl-Encode-2.80 breaks sending email by bugzilla
perl-Encode-2.80 breaks sending email by bugzilla
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: bugzilla (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Emmanuel Seyman
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-05 08:50 EST by Sjoerd Mullender
Modified: 2016-05-28 19:54 EDT (History)
8 users (show)

See Also:
Fixed In Version: bugzilla-5.0.2-3.fc24 bugzilla-4.4.12-1.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-28 19:54:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
possible patch to fix the problem (790 bytes, patch)
2016-04-07 15:18 EDT, Sjoerd Mullender
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Mozilla Foundation 1246228 None None None 2016-05-02 03:08 EDT
CPAN 111853 None None None 2016-02-08 04:04 EST

  None (edit)
Description Sjoerd Mullender 2016-02-05 08:50:44 EST
Description of problem:
I don't know whether this bug is in bugzilla or perl-Encode, but since the update to perl-Encode-2.80-4.fc23.x86_64 my bugzilla installation produces emails that are horrible.  I suspect the this commit:
https://github.com/dankogai/p5-encode/commit/77c0a92d8d12f06a3d92ea3d798ec81170b9202b
Mails sent by bugzilla contain a Date header that my thunderbird can't parse, the To address is mangled, and all headers that bugzilla creates use quoted-printable encoding.
An example:
To: =?UTF-8?Q?bugs=2Dsql=40monetdb=2Eorg?=@monetdb.org
Subject: =?UTF-8?Q?=5BBug=203916=5D=20New=3A=20consideration=20for=20MonetDBLite=3A=20manage=20database=20updates=20properly?=
Date: =?UTF-8?Q?Thu=2C=2004=20Feb=202016=2011=3A02=3A14=20=2B0000?=
X-Bugzilla-Reason: =?UTF-8?Q?AssignedTo?=

Note that the decoded version of the To header has two @ signs.

Version-Release number of selected component (if applicable):
perl-Encode-2.80-4.fc23.x86_64

How reproducible:
100%

Steps to Reproduce:
1.have bugzilla send an email
2.
3.

Actual results:
See above.  All headers that bugzilla produces (as opposed to the ones the MTA adds) use quoted-printable encoding, even when they are fully ASCII.

Expected results:
No quoted-printable headers.


Additional info:
A perl-savvy person can probably just call the encode function directly and doesn't have to go through bugzilla.  https://rt.cpan.org/Public/Bug/Display.html?id=88717 mentions
print encode('MIME-Header', "Hey foo\x{2024}bar:whee")."\n";
I'd try something with pure ASCII as second argument.  The result of that should, in my oppinion, not use quoted-printable encoding.
Comment 1 Petr Pisar 2016-02-08 04:04:09 EST
(In reply to Sjoerd Mullender from comment #0)
> Description of problem:
> I don't know whether this bug is in bugzilla or perl-Encode, but since the
> update to perl-Encode-2.80-4.fc23.x86_64 my bugzilla installation produces
> emails that are horrible.  I suspect the this commit:
> https://github.com/dankogai/p5-encode/commit/
> 77c0a92d8d12f06a3d92ea3d798ec81170b9202b
> Mails sent by bugzilla contain a Date header that my thunderbird can't
> parse, the To address is mangled, and all headers that bugzilla creates use
> quoted-printable encoding.
> An example:
> To: =?UTF-8?Q?bugs=2Dsql=40monetdb=2Eorg?=@monetdb.org
> Subject:
> =?UTF-
> 8?Q?=5BBug=203916=5D=20New=3A=20consideration=20for=20MonetDBLite=3A=20manage
> =20database=20updates=20properly?=
> Date: =?UTF-8?Q?Thu=2C=2004=20Feb=202016=2011=3A02=3A14=20=2B0000?=
> X-Bugzilla-Reason: =?UTF-8?Q?AssignedTo?=
> 
Could you please describe what the data looks before encoding? Without knowing the input, I cannot know what the correct output should be expected.

> All headers that bugzilla produces (as opposed to the ones the MTA adds)
> use quoted-printable encoding, even when they are fully ASCII.

This has been reported to upstream as <https://rt.cpan.org/Public/Bug/Display.html?id=111853>. It looks ugly, but it does not violate any specification. If Bugzilla does not want to encode the e-mail addresses, then it should not submit them to MIME-encoding function.
Comment 2 Sjoerd Mullender 2016-02-08 04:18:33 EST
(In reply to Petr Pisar from comment #1)
> (In reply to Sjoerd Mullender from comment #0)
> > Description of problem:
> > I don't know whether this bug is in bugzilla or perl-Encode, but since the
> > update to perl-Encode-2.80-4.fc23.x86_64 my bugzilla installation produces
> > emails that are horrible.  I suspect the this commit:
> > https://github.com/dankogai/p5-encode/commit/
> > 77c0a92d8d12f06a3d92ea3d798ec81170b9202b
> > Mails sent by bugzilla contain a Date header that my thunderbird can't
> > parse, the To address is mangled, and all headers that bugzilla creates use
> > quoted-printable encoding.
> > An example:
> > To: =?UTF-8?Q?bugs=2Dsql=40monetdb=2Eorg?=@monetdb.org
> > Subject:
> > =?UTF-
> > 8?Q?=5BBug=203916=5D=20New=3A=20consideration=20for=20MonetDBLite=3A=20manage
> > =20database=20updates=20properly?=
> > Date: =?UTF-8?Q?Thu=2C=2004=20Feb=202016=2011=3A02=3A14=20=2B0000?=
> > X-Bugzilla-Reason: =?UTF-8?Q?AssignedTo?=
> > 
> Could you please describe what the data looks before encoding? Without
> knowing the input, I cannot know what the correct output should be expected.

To: bugs-sql@monetdb.org@monetdb.org
Subject: [Bug 3916] New: consideration for MonetDBLite: manage database updates properly
Date: Thu, 04 Feb 2016 11:02:14 +0000
X-Bugzilla-Reason: AssignedTo

> > All headers that bugzilla produces (as opposed to the ones the MTA adds)
> > use quoted-printable encoding, even when they are fully ASCII.
> 
> This has been reported to upstream as
> <https://rt.cpan.org/Public/Bug/Display.html?id=111853>. It looks ugly, but
> it does not violate any specification. If Bugzilla does not want to encode
> the e-mail addresses, then it should not submit them to MIME-encoding
> function.

That why I wrote that I didn't know whether this bug is in Bugzilla or perl-Encode.  ;-)

I don't know what the API is, but it seems wrong if the Date and To fields get mangled the way they were.  The To field given to perl-Encode was the address bugs-sql@monetdb.org, this was quoted by perl-Encode to =?UTF-8?Q?bugs=2Dsql=40monetdb=2Eorg?= which was then not recognized by the MTA as a fully qulified email address, so it added a domain, hence the two @ signs in the deocded result.
Comment 3 Petr Pisar 2016-03-24 06:05:30 EDT
Upstream completely rewrote the MIME coding in Encode-2.83. The old code was one big mess with corner cases violating RFC 2047. I will check if it could help you.
Comment 4 Petr Pisar 2016-03-24 07:27:20 EDT
2.83 still encodes everything.
Comment 5 Sjoerd Mullender 2016-04-07 15:18 EDT
Created attachment 1144849 [details]
possible patch to fix the problem

Here is a patch that I hope fixes the problem.  I haven't tried it out for real, but the reported failure when I run "make test" looks like it does the right thing.

What the code did before I intervened is to take the whole header value (i.e. everything after the first colon) and encode it in its entirety as UTF-8 quoted printable.  What I did instead is to take that complete value, split it on white space, and for each chunk check whether it is pure ASCII (range ! to ~) and if so, pass it unchanged.  Otherwise use the old code to encode the piece.  Then at the end, join all the pieces (encoded and unchanged) with a space between them.

A proper Perl programmer might want to take a look whether this is any good.

The code is based on the patched source from perl-Encode-2.80-4.fc23.src.rpm (i.e., I ran rpmbuild -bp).
Comment 6 Petr Pisar 2016-04-08 02:59:20 EDT
Splitting on spaces and then concatenating is problematic. ASCI SPACE QUOTED sequence and QUOTED SPACE QUOTED sequence has different rules for ignoring the SPACE in between.

The Encode::MIME::Header was broken because of this and the 2.83 fixed it finally. The resolution is that Encode::MIME::Header will always encode whole header value as one chunk. If an application wants to encode headers with e-mail addresses (e.g. To:), it must do the segmentation by itself and pass for encoding only the interested parts. See <https://rt.cpan.org/Public/Bug/Display.html?id=111853> and a fix for a broken Encode::MIME::Header user <https://github.com/theory/svn-notify/pull/15>.

I don't know yet what to do in Fedora 23. Maybe I will applyr you patch to restore older behavior in the expense that the encoding will be broken in some cases.

But Fedora >= 24 will include the 2.83 version that does not do the segmentation and the one who must be fixed there will be the applications.

I'm sorry upgrading the perl-Encode in Fedora 23 caused problems to you, I will try to restore the old behavior thankfully to your patch.

Please talk to Encode developers in <https://rt.cpan.org/Public/Bug/Display.html?id=111853>. They are experts in this area and I believe they can explain the Encode design better than me and can help you with developing the best fix for you applications.
Comment 7 Sjoerd Mullender 2016-04-08 03:30:11 EDT
As I wrote in the original bug report, I didn't know whether the problem was in perl-Encode or bugzilla.  It's the combination of the two that causes the headaches.  If this is not actually a bug in perl-Encode, then this is a bug in bugzilla.  It should do the splitting before handing the headers over to perl-Encode.
Comment 8 Petr Pisar 2016-04-08 04:47:56 EDT
I tried your patch. All the places where Encode-2.80 tests fail are expected except this one:

To: dankogai@dan.co.jp (小飼=Kogai, 弾=Dan)

The original 2.80 code produces:

To: =?UTF-8?Q?dankogai=40dan=2Eco=2Ejp=20=28?=
 =?UTF-8?Q?=E5=B0=8F=E9=A3=BC=3DKogai=2C=20=E5=BC=BE=3DDan?= =?UTF-8?Q?=29?=

The patched 2.80 code produces:

To: dankogai@dan.co.jp (
 =?UTF-8?Q?=E5=B0=8F=E9=A3=BC=3DKogai=2C?= =?UTF-8?Q?=E5=BC=BE=3DDan?=)

That means it inserts a space between "(" and "小". This is because splitting the text into multiple lines happens before calling _encode_q(). The line splitting code expects all the lines will start and end with encoded-word (=?...?=) so the line break will be ignored when decoding.

But in this case you have a plain line followed by an encoded line, and in this case the line break is decoded as a space.

I feel it's not possible to fix Encode-2.80 to restore original behavior without introducing bugs.

If your Bugzilla installation comes from Fedora's package, please file a bug against bugzilla component to fix how Bugzilla uses Encode::encode('MIME-Q').
Comment 9 Sjoerd Mullender 2016-04-08 05:24:22 EDT
I don't agree with the resolution.  It *is* a bug.  Maybe not in perl-Encode, but then in bugzilla.  So this bug report should then be assigned to the bugzilla component.
Comment 10 Michael Tremer 2016-04-08 17:16:30 EDT
I can confirm that this is a bug with F23 and the latest shipped Bugzilla release 4.4.11-1.fc23. I upgraded to bugzilla-5.0.2-2.fc24 and the issue persists.

A downgrade to perl-Encode-2.78-2.fc23 fixes the problem.

I would like to ask you to re-open this bug and assign to someone who is either familiar with perl-Encode and/or Bugzilla.
Comment 11 Sjoerd Mullender 2016-04-08 17:42:57 EDT
Reopening and assigning this to bugzilla.
Comment 12 Michael Cronenworth 2016-04-29 10:23:25 EDT
perl-Encode 2.79 is the last version that worked with Bugzilla. Version 2.80 and up to 2.84 are broken.
Comment 13 Emmanuel Seyman 2016-04-29 16:15:11 EDT
From the #bugzilla IRC channel:

21:08 <@dylan> eseyman: yes, that is a known bug in that module
21:08 <@dylan> we will have to work around it, the best solution is downgrading it at the moment
Comment 14 Frédéric Buclin 2016-04-30 18:10:47 EDT
(In reply to Emmanuel Seyman from comment #13)
> 21:08 <@dylan> eseyman: yes, that is a known bug in that module
> 21:08 <@dylan> we will have to work around it, the best solution is
> downgrading it at the moment

This is not a bug in perl-Encode. It was a bug in the Bugzilla code itself which has been fixed a month ago, see:

https://bugzilla.mozilla.org/show_bug.cgi?id=1246228
https://git.mozilla.org/?p=bugzilla/bugzilla.git;a=commitdiff;h=652ee918

This fix will be in Bugzilla 5.0.3.
Comment 15 Fedora Update System 2016-05-02 04:08:52 EDT
bugzilla-5.0.2-3.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3527028bbe
Comment 16 Sjoerd Mullender 2016-05-02 04:10:44 EDT
Is there also going to come a fix for Fedora 23?
Comment 17 Emmanuel Seyman 2016-05-02 04:17:49 EDT
I've applied the patch in rawhide and issued an update for f24.
@lpsolit, is there a reason Bugzilla's 4.4 branch didn't receive the fix?
Comment 18 Fedora Update System 2016-05-02 06:53:01 EDT
bugzilla-5.0.2-3.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-3527028bbe
Comment 19 Fedora Update System 2016-05-11 21:31:53 EDT
bugzilla-5.0.2-3.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Comment 20 Sjoerd Mullender 2016-05-12 02:32:53 EDT
It's nice that the bug is fixed on Fedora 24, but I'm running Fedroa 23, and I reported the bug for Fedora 23.  So I have to reopen this.
Comment 21 Fedora Update System 2016-05-12 06:05:51 EDT
bugzilla-4.4.11-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-bcffbf3c08
Comment 22 Emmanuel Seyman 2016-05-12 06:11:32 EDT
(In reply to Sjoerd Mullender from comment #20)
>
> It's nice that the bug is fixed on Fedora 24, but I'm running Fedroa 23, and
> I reported the bug for Fedora 23.  So I have to reopen this.

Indeed. I thought putting the bug back to the ASSIGNED status would prevent bodhi from touching it again but it looks like I was wrong.

Okay, given the lack of activity of bmo's bug #1246228, I've decided to include Frédéric's submitted patch. Sjoerd, testing the update I just put out would be very appreciated (it works for me but it's more important that it works for you).
Comment 23 Sjoerd Mullender 2016-05-12 06:45:43 EDT
Testing will have to wait until (well) after the weekend.  Monday is a holiday here, and we're making use of it.
Comment 24 Petr Pisar 2016-05-12 07:06:20 EDT
Thank you for fixing the Bugzilla.
Comment 25 Fedora Update System 2016-05-13 01:31:58 EDT
bugzilla-4.4.11-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-bcffbf3c08
Comment 26 Fedora Update System 2016-05-17 18:10:39 EDT
bugzilla-4.4.12-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6cdcddef2c
Comment 27 Fedora Update System 2016-05-20 22:28:40 EDT
bugzilla-4.4.12-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6cdcddef2c
Comment 28 Fedora Update System 2016-05-28 19:54:41 EDT
bugzilla-4.4.12-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.