Bug 603635

+++ This bug was initially created as a clone of Bug #461707 +++

* Description of problem:
When customer send an email in certain way, CC filed in
received email shows wiered output.

For example, If email is sent in the following strings,

To: test-list
Cc: DS◯◯半導体 ◯◯課長1 <test1.com>
Cc: DS◯◯半導体 ◯◯課長2 <test1.com>

Output of Cc field in received email becomes as follows.

To: test-list
Cc: 長2, DS◯◯半導体 ◯◯課長,DS◯◯半導体 ◯◯課.com

This problem occurs only when a mailing list created in mailman is specified in To/Cc field as one send an email. So this seems to be mailman specific problem.   


* How reproducible:
Always.


*Steps to Reproduce:

1. On one machine(eg, node1.example.com), setup postfix so
  that it can send mail to each user within the local
  machine and setup mailman and create one
  list(eg,test-list).

2. Create at least 3 users for mail accounts(eg,user1,user2,
  user3) and include user1 into the list(test-list).

3. Send a mail from user2 or user3 account in the following
  way.

  To: test-list.com
  Cc: DS◯◯半導体 ◯◯課長1 <user2.com>
  Cc: DS◯◯半導体 ◯◯課長2 <user3.com>

4. Check the mail in user1 account(user1 should have received
  the mail through the test-list) and see Cc field on the
  received mail. It should look like the following.

  From: user2(or user3)
  Sender: test-list-bounces.com
  Date: XX:XX XM
  To: test-list.com
  Cc: 長2,◯◯半導体 ◯◯課長1,DS◯◯半導体 ◯◯課.com

Note: This problem is reproduceable even if you use only
     English charactors. I couldn't get what pattern of
     string it occurs. But looks like it occurs when you put
     long strings in Cc field. Here is example of step 3 in
     English charactors.

3. Send a mail from user2 or user3 account in the following
  way.
  
 To: test-list.com
 Cc: DSDS◯◯ handotai ◯◯kachokacho2 <user2.node1.example.com>
 Cc: DSDS◯◯ handotai ◯◯kachokacho3 <user3.node1.example.com>

4. Check the mail in user1 account(user1 should have
  received the mail through the test-list) and see Cc field
  on the received mail. It should look like the following.

  achokacho3,DSDS◯◯ handotai ◯◯kachokacho2,DSDS◯◯ handotai ◯◯k.com

*The display in Cc field might look different depending on if you sent the mail in UTF-8 or other charactor set. It does look wierd anyway though.  


*Actual results:
Output in Cc field in received mail looks wierd.


*Expected results:
Display name should be printed in Cc field correctly.


*Additional info:
mailman-2.1.9-2/ RHEL5.1

----------------------------------

The problem is that python script in mailman put COMMASPACE(", ") in between strings in Cc field where it shouldn't. By putting COMMASPACE in between strings, it'll split the strings in Cc field on the display of email clients.

*CC field in Recieved mail header

Cc: =?ISO-2022-JP?B?GyRCIXkheSF5IXkheSF5IXkheSF5IXkheSF5IXkheSF5IXkheRsoQg==?=@dhcp-1-164.bne.redhat.com,
       =?ISO-2022-JP?B?GyRCIXkheSF5GyhC?= <user5.redhat.com>

Note: COMMASPACE(", ") after @dhcp-1-164.bne.redhat.com is causing the problem.

The code adding the COMMPASPACE(", ") in the Cc field is the following.

From /usr/lib/mailman/pythonlib/email/Utils.py

115 def getaddresses(fieldvalues):
116     """Return a list of (REALNAME, EMAIL) for each fieldvalue."""
117     all = COMMASPACE.join(fieldvalues)
118     a = _AddressList(all)
119     return a.addresslist

I've changed the line:117 as follows for testing. This way, it doesn't put COMMASPACE in between the strings.

117     all = fieldvalues

After I've made the above change, I couldn't observe the problem any longer. Even if I put multiple recipients in Cc field. But I doubt this is the right method


getaddresses is used in /usr/lib/mailman/Mailman/Handlers/AvoidDuplicates.py

def process(mlist, msg, msgdata):
   ...
   # RFC 2822 specifies zero or one CC header
   del msg['cc']
   if ccaddrs:
       msg['Cc'] = COMMASPACE.join([formataddr(i) for i in ccaddrs.values()])

--- Additional comment from tao on 2008-10-29 20:39:58 EDT ---

Hi,

It's been few months since I've escalated this ticket. My customer is
continuously contacting me for the fix.

Could you give the current status of this issue? The customer especially
wonder when it will be fixed so that they can plan the deployment of the
system.

Please let me know which update you're planning to fix the issue.

Regards,

Masahiro


This event sent from IssueTracker by mokubo 
 issue 184316

--- Additional comment from dnovotny on 2008-11-06 04:44:53 EST ---

hello,

I am a new maintainer of mailman and I've started to look at this bug. From the analysis you showed above I think there is a different problem:

 def getaddresses(fieldvalues):
116     """Return a list of (REALNAME, EMAIL) for each fieldvalue."""
117     all = COMMASPACE.join(fieldvalues)

the problem is, that "fieldvalues" should contain a list, where each item will be *one mail address* - that way, when you put COMMASPACE between them, you will get addresses separated by commas.

in the example of the errorneous behavior above it is clear, that the *splitting* is done wrong: it splits inbetween, so one adress is teared apart and mailman thinks they are two (and puts COMMASPACE between "them")

the question is, where does this splitting occur: if you send wrong "fieldvalues" -splitted inbetween- to getaddresses(), you will get wrong cc field, obviously

so the analysis have to look, why the caller of the function "getaddresses()" puts two separated items in the list, where there should be one...

--- Additional comment from pm-rhel on 2009-03-26 13:27:27 EDT ---

This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".

--- Additional comment from dnovotny on 2009-06-12 08:53:53 EDT ---

Created an attachment (id=347555)
this is the sample of the mail which causes the problem

--- Additional comment from dnovotny on 2009-06-12 08:58:15 EDT ---

looking at the sample mail (comment #6)
I can see, that the name is broken apart by a newline character, because it is long... this seems to be parsed by _parseaddr.py . The investigation led me to a similar Debian bug:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=516300

they solved the problem by patching the _parseaddr.py, backporting from a newer version:

--- /usr/lib/mailman/pythonlib/email/_parseaddr.py.distrib	2006-06-13 05:43:49.000000000 +0200
+++ /usr/lib/mailman/pythonlib/email/_parseaddr.py	2009-02-20 13:19:35.000000000 +0100
@@ -170,6 +170,7 @@
         self.pos = 0
         self.LWS = ' \t'
         self.CR = '\r\n'
+        self.FWS = self.LWS + self.CR
         self.atomends = self.specials + self.LWS + self.CR
         # Note that RFC 2822 now specifies `.' as obs-phrase, meaning that it
         # is obsolete syntax.  RFC 2822 requires that we recognize obsolete
@@ -416,7 +417,7 @@
         plist = []
 
         while self.pos < len(self.field):
-            if self.field[self.pos] in self.LWS:
+            if self.field[self.pos] in self.FWS:
                 self.pos += 1
             elif self.field[self.pos] == '"':
                 plist.append(self.getquote())

--- Additional comment from dnovotny on 2009-06-12 08:59:37 EDT ---

can you try this, if it helps our problem?

--- Additional comment from dnovotny on 2009-06-12 09:06:48 EDT ---

Created an attachment (id=347558)
the proposed patch

--- Additional comment from tao on 2009-06-22 20:13:51 EDT ---

Event posted on 06-22-2009 08:13pm EDT by mokubo

Hi,

Any update on this? Our customer want the fix as early as possible and
wants to know when the fix
will be included.

Please let us know when the patch will be included in the package.

Regards,

Masa





This event sent from IssueTracker by mokubo 
 issue 184316

--- Additional comment from dnovotny on 2009-06-24 06:29:20 EDT ---

hello Masa,

with my patch, the cc field seems to be all in one row and not broken, but I'm not sure if I reproduced the problem correctly, because I don't know Japanese characters. I've built RPM packages of mailman with my patch, so the customer (or you) can use them and see if this helps

http://people.fedoraproject.org/~dnovotny/ccfield


 Daniel