Bug 1288718 - policyd-spf issuing traceback on emails from bilbo2.documentfoundation.org
Summary: policyd-spf issuing traceback on emails from bilbo2.documentfoundation.org
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-ipaddress
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Nathaniel McCallum
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-05 14:51 UTC by Trever Adams
Modified: 2017-01-29 00:49 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-29 00:20:05 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Trever Adams 2015-12-05 14:51:53 UTC
Description of problem:
Some emails sent through the libreoffice mailing list cause policyd-spf to issue a traceback causing emails to be rejected with 451 Configuation error. I am currently only seeing this with SOME emails from libreoffice mailing list and so far no where else.

Version-Release number of selected component (if applicable):
pypolicyd-spf-1.3.1-3.fc23.noarch

How reproducible:
Not sure. Not all through the list do it.

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Dec  5 07:05:01 SystemA postfix/postscreen[3995]: CONNECT from [2a01:4f8:190:3144::2]:40800 to [2001:470:1f11:1007::1]:25
Dec  5 07:05:01 SystemA postfix/postscreen[3995]: PASS OLD [2a01:4f8:190:3144::2]:40800
Dec  5 07:05:01 SystemA postfix/smtpd[3996]: connect from bilbo2.documentfoundation.org[2a01:4f8:190:3144::2]
Dec  5 07:05:02 SystemA postfix/smtpd[3996]: Anonymous TLS connection established from bilbo2.documentfoundation.org[2a01:4f8:190:3144::2]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Dec  5 07:05:03 SystemA policyd-spf[4002]: Traceback (most recent call last):
Dec  5 07:05:03 SystemA policyd-spf[4002]:  File "/usr/libexec/postfix/policyd-spf", line 687, in <module>
Dec  5 07:05:03 SystemA policyd-spf[4002]:    instance_dict, configData, peruser)
Dec  5 07:05:03 SystemA policyd-spf[4002]:  File "/usr/libexec/postfix/policyd-spf", line 420, in _spfcheck
Dec  5 07:05:03 SystemA policyd-spf[4002]:    res = spf.check2(ip, helo_fake_sender, helo, querytime=configData.get('Lookup_Time'))
Dec  5 07:05:03 SystemA policyd-spf[4002]:  File "/usr/lib/python2.7/site-packages/spf.py", line 290, in check2
Dec  5 07:05:03 SystemA policyd-spf[4002]:    receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
Dec  5 07:05:03 SystemA policyd-spf[4002]:  File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check
Dec  5 07:05:03 SystemA policyd-spf[4002]:    rc = self.check1(spf, self.d, 0)
Dec  5 07:05:03 SystemA policyd-spf[4002]:  File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1
Dec  5 07:05:03 SystemA policyd-spf[4002]:    return self.check0(spf, recursion)
Dec  5 07:05:03 SystemA policyd-spf[4002]:  File "/usr/lib/python2.7/site-packages/spf.py", line 891, in check0
Dec  5 07:05:03 SystemA policyd-spf[4002]:    if self.cidrmatch(self.dns_a(arg,self.A), cidrlength):
Dec  5 07:05:03 SystemA policyd-spf[4002]:  File "/usr/lib/python2.7/site-packages/spf.py", line 1202, in dns_a
Dec  5 07:05:03 SystemA policyd-spf[4002]:    return [ipaddress.Bytes(ip) for ip in r]
Dec  5 07:05:03 SystemA policyd-spf[4002]: AttributeError: 'module' object has no attribute 'Bytes'
Dec  5 07:05:03 SystemA postfix/spawn[4001]: warning: command /usr/libexec/postfix/policyd-spf exit status 1
Dec  5 07:05:03 SystemA postfix/smtpd[3996]: warning: premature end-of-input on private/policy-spf while reading input attribute name
Dec  5 07:05:04 SystemA policyd-spf[4004]: Traceback (most recent call last):
Dec  5 07:05:04 SystemA policyd-spf[4004]:  File "/usr/libexec/postfix/policyd-spf", line 687, in <module>
Dec  5 07:05:04 SystemA policyd-spf[4004]:    instance_dict, configData, peruser)
Dec  5 07:05:04 SystemA policyd-spf[4004]:  File "/usr/libexec/postfix/policyd-spf", line 420, in _spfcheck
Dec  5 07:05:04 SystemA policyd-spf[4004]:    res = spf.check2(ip, helo_fake_sender, helo, querytime=configData.get('Lookup_Time'))
Dec  5 07:05:04 SystemA policyd-spf[4004]:  File "/usr/lib/python2.7/site-packages/spf.py", line 290, in check2
Dec  5 07:05:04 SystemA policyd-spf[4004]:    receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
Dec  5 07:05:04 SystemA policyd-spf[4004]:  File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check
Dec  5 07:05:04 SystemA policyd-spf[4004]:    rc = self.check1(spf, self.d, 0)
Dec  5 07:05:04 SystemA policyd-spf[4004]:  File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1
Dec  5 07:05:04 SystemA policyd-spf[4004]:    return self.check0(spf, recursion)
Dec  5 07:05:04 SystemA policyd-spf[4004]:  File "/usr/lib/python2.7/site-packages/spf.py", line 891, in check0
Dec  5 07:05:04 SystemA policyd-spf[4004]:    if self.cidrmatch(self.dns_a(arg,self.A), cidrlength):
Dec  5 07:05:04 SystemA policyd-spf[4004]:  File "/usr/lib/python2.7/site-packages/spf.py", line 1202, in dns_a
Dec  5 07:05:04 SystemA policyd-spf[4004]:    return [ipaddress.Bytes(ip) for ip in r]
Dec  5 07:05:04 SystemA policyd-spf[4004]: AttributeError: 'module' object has no attribute 'Bytes'
Dec  5 07:05:04 SystemA postfix/spawn[4001]: warning: command /usr/libexec/postfix/policyd-spf exit status 1
Dec  5 07:05:04 SystemA postfix/smtpd[3996]: warning: premature end-of-input on private/policy-spf while reading input attribute name
Dec  5 07:05:04 SystemA postfix/smtpd[3996]: warning: problem talking to server private/policy-spf: Success

I imagine some of the code here needs to be a in a try block or some other error prevention/catching method.

Comment 1 Bojan Smojver 2015-12-06 05:47:17 UTC
The problem is that with a switch to python-ipaddress instead of python-ipaddr, some stuff appears to be missing from the API. So, python-pyspf will need to be patched again to fix this.

Comment 2 Bojan Smojver 2015-12-09 00:24:14 UTC
Total guess here (I'm not a python guy at all), but does this patch do anything for you?

----------------------
diff -ruN pyspf-2.0.11-v/spf.py pyspf-2.0.11/spf.py
--- pyspf-2.0.11-v/spf.py	2014-12-06 03:20:07.000000000 +1100
+++ pyspf-2.0.11/spf.py	2015-12-08 11:41:35.025136287 +1100
@@ -1203,7 +1203,7 @@
                     'No %s records found for'%A, domainname)
         if A == 'AAAA' and bytes is str:
           # work around pydns inconsistency plus python2 bytes/str ambiguity
-          return [ipaddress.Bytes(ip) for ip in r]
+          return [Bytes(ip) for ip in r]
         return r
 
     def validated_ptrs(self):
@@ -1617,6 +1617,22 @@
               % (sender, self.c)
         raise ValueError("invalid SPF result for header comment: "+res)
 
+# We need to distinguish between the string and packed-bytes representations
+# of an IP address.  For example, b'0::1' is the IPv4 address 48.58.58.49,
+# while '0::1' is an IPv6 address.
+#
+# In Python 3, the native 'bytes' type already provides this functionality,
+# so we use it directly.  For earlier implementations where bytes is not a
+# distinct type, we create a subclass of str to serve as a tag.
+try:
+    if bytes is str:
+        raise TypeError("bytes is not a distinct type")
+    Bytes = bytes
+except (NameError, TypeError):
+    class Bytes(str):
+        def __repr__(self):
+            return 'Bytes(%s)' % str.__repr__(self)
+
 def split_email(s, h):
     """Given a sender email s and a HELO domain h, create a valid tuple
     (l, d) local-part and domain-part.
----------------------

Comment 3 Trever Adams 2015-12-09 21:01:58 UTC
This patch seems to break all IPv6 connections as I am not getting the error between two domains that have the same configuration (I do not believe it was there before) and with ack.nmap.org.

Comment 4 Bojan Smojver 2015-12-09 21:09:32 UTC
(In reply to Trever Adams from comment #3)
> This patch seems to break all IPv6 connections as I am not getting the error
> between two domains that have the same configuration (I do not believe it
> was there before) and with ack.nmap.org.

Thanks for testing and sorry about the failure. I have absolutely no idea what I'm doing there. I think someone with the knowledge of python and the code in python-pyspf will need to have a look.

Comment 5 Trever Adams 2015-12-10 13:06:41 UTC
With the patch the traceback changes.

Dec  9 15:13:53 SystemA policyd-spf[25731]: None; identity=helo; client-ip=2600:3c01::f03c:91ff:fe98:ff4e; helo=ack.nmap.org; envelope-from=fulldisclosure-bounces; receiver=trever.org
Dec  9 15:13:53 SystemA policyd-spf[25731]: Traceback (most recent call last):
Dec  9 15:13:53 SystemA policyd-spf[25731]:  File "/usr/libexec/postfix/policyd-spf", line 687, in <module>
Dec  9 15:13:53 SystemA policyd-spf[25731]:    instance_dict, configData, peruser)
Dec  9 15:13:53 SystemA policyd-spf[25731]:  File "/usr/libexec/postfix/policyd-spf", line 524, in _spfcheck
Dec  9 15:13:53 SystemA policyd-spf[25731]:    res = spf.check2(ip, sender, helo, querytime=configData.get('Lookup_Time'))
Dec  9 15:13:53 SystemA policyd-spf[25731]:  File "/usr/lib/python2.7/site-packages/spf.py", line 290, in check2
Dec  9 15:13:53 SystemA policyd-spf[25731]:    receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
Dec  9 15:13:53 SystemA policyd-spf[25731]:  File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check
Dec  9 15:13:53 SystemA policyd-spf[25731]:    rc = self.check1(spf, self.d, 0)
Dec  9 15:13:53 SystemA policyd-spf[25731]:  File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1
Dec  9 15:13:53 SystemA policyd-spf[25731]:    return self.check0(spf, recursion)
Dec  9 15:13:53 SystemA policyd-spf[25731]:  File "/usr/lib/python2.7/site-packages/spf.py", line 891, in check0
Dec  9 15:13:53 SystemA policyd-spf[25731]:    if self.cidrmatch(self.dns_a(arg,self.A), cidrlength):
Dec  9 15:13:53 SystemA policyd-spf[25731]:  File "/usr/lib/python2.7/site-packages/spf.py", line 1347, in cidrmatch
Dec  9 15:13:53 SystemA policyd-spf[25731]:    ipaddrs[idx] = ipaddrs[idx].decode('ascii')
Dec  9 15:13:53 SystemA policyd-spf[25731]: UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 8: ordinal not in range(128)
Dec  9 15:13:53 SystemA postfix/spawn[25191]: warning: command /usr/libexec/postfix/policyd-spf exit status 1
Dec  9 15:13:53 SystemA postfix/smtpd[25551]: warning: premature end-of-input on private/policy-spf while reading input attribute name
Dec  9 15:13:53 SystemA postfix/smtpd[25551]: warning: problem talking to server private/policy-spf: Success
Dec  9 15:13:53 SystemA postfix/smtpd[25551]: NOQUEUE: reject: RCPT from ack.nmap.org[2600:3c01::f03c:91ff:fe98:ff4e]: 451 4.3.5 <trever.org>: Recipient address rejected: Server configuration problem; from=<fulldisclosure-bounces> to=<trever.org> proto=ESMTP helo=<ack.nmap.org>

Comment 6 Trever Adams 2015-12-10 13:07:42 UTC
Oh, and it is still not 100% consistent. Is it possible this is being caused by dns lookups being a bit slow at times? (My CenturyLink pppoe connection is not the best and stalls at times due to them refusing to fix some problems with the line.)

Comment 7 Bojan Smojver 2015-12-11 02:34:26 UTC
OK, one last try before I give up:
-------------------
diff -ruN pyspf-2.0.11-v/spf.py pyspf-2.0.11/spf.py
--- pyspf-2.0.11-v/spf.py	2014-12-06 03:20:07.000000000 +1100
+++ pyspf-2.0.11/spf.py	2015-12-11 13:31:46.102844734 +1100
@@ -1201,9 +1201,6 @@
         if self.strict > 1 and len(r) == 0:
             raise AmbiguityWarning(
                     'No %s records found for'%A, domainname)
-        if A == 'AAAA' and bytes is str:
-          # work around pydns inconsistency plus python2 bytes/str ambiguity
-          return [ipaddress.Bytes(ip) for ip in r]
         return r
 
     def validated_ptrs(self):
-------------------

Essentially, don't touch what's returned, but just pass it on. Absolutely no idea whether this will screw things up even more.

Comment 8 Trever Adams 2015-12-11 16:04:21 UTC
Is this in lieu of or in addition to the other patch?

Comment 9 Bojan Smojver 2015-12-11 21:31:26 UTC
Just on its own.

Comment 10 Trever Adams 2015-12-12 14:24:42 UTC
It does not solve the problem. Whatever it is is transient. Messages get rejected and then accepted later. I have no clue what is going on other than the traceback. IPv6 seems to be the only commonality.

Comment 11 Bojan Smojver 2015-12-13 01:37:57 UTC
I'm going to assign this to python-ipaddress package now, so that folks maintaining that can point us in the right direction. I have no idea how this is supposed to work, what should play the role of ipaddress.Bytes here etc.

Comment 12 Trever Adams 2016-05-21 23:11:51 UTC
This problem still exists in fc24 with the following packages.
python-ipaddress-1.0.16-2.fc24.noarch
python-pyspf-2.0.11-4.fc24.noarch

Comment 13 Trever Adams 2016-05-24 14:40:15 UTC
Unfortunately, my DSL provider is a bit flaky at times. 90% of the problems happen within 1 minute of the ppp connection resetting. The rest of the time, I am assuming the connection has slowed enough to cause a timeout of the dns request.

So, this problem isn't a bug in the normal code path, but either a bug in an error path or the lack thereof.

Comment 14 Trever Adams 2016-07-08 14:23:11 UTC
After another month of watching, yes, the connection dropping and therefore DNS problems will cause this, but that isn't it. the address listed in the title of this report yields IPv6 results. It happens very consistently. If it comes in over IPv4 there is no problem.

Comment 15 Trever Adams 2016-11-28 16:03:14 UTC
It appears that it happens far more commonly with IPv6 addresses with ::.

Comment 16 Bojan Smojver 2016-11-29 00:39:12 UTC
I've just  built this in F26:

http://koji.fedoraproject.org/koji/buildinfo?buildID=820968

I haven't tested it at all, but it is python3 based. Maybe that will be better with IPv6 addresses. No idea.

Anyhow, there it is, in case you want to try it.

Comment 17 Trever Adams 2016-11-29 00:48:07 UTC
I am not seeing the required python3-pyspf. Is that built somewhere?

Comment 18 Trever Adams 2016-11-29 00:53:51 UTC
My mistake, koji only lists it under python-pyspf.

Comment 19 Trever Adams 2016-11-30 03:57:12 UTC
This may have fixed it. I got a bunch from the documentfoundation mailing list today with no problems. I will give it a few more days before I say it is fixed. If it is, can we see this in F25?

Comment 20 Bojan Smojver 2016-11-30 04:47:43 UTC
(In reply to Trever Adams from comment #19)
> If it is, can we see this in F25?

Absolutely. And F24, if all goes as planned with pyspf maintainers.

Comment 21 Fedora Update System 2016-12-01 03:17:48 UTC
pypolicyd-spf-1.3.2-4.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c

Comment 22 Fedora Update System 2016-12-01 03:18:01 UTC
pypolicyd-spf-1.3.2-4.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27

Comment 23 Trever Adams 2016-12-02 20:45:52 UTC
Thank you Bojan, I can comfortably confirm that the bug has been fixed by moving to pure python3. Thank you! Please, feel free to close this bug as fixed with the packages in question when all parties have pushed their packages.

Thank you all!

Comment 24 Bojan Smojver 2016-12-02 23:47:27 UTC
(In reply to Trever Adams from comment #23)
> Thank you Bojan, I can comfortably confirm that the bug has been fixed by
> moving to pure python3. Thank you! Please, feel free to close this bug as
> fixed with the packages in question when all parties have pushed their
> packages.
> 
> Thank you all!

Thank you for testing. The bug will be automatically closed when the updated packages reach stable.

Comment 25 Fedora Update System 2016-12-03 04:34:21 UTC
pypolicyd-spf-1.3.2-4.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c

Comment 26 Fedora Update System 2016-12-03 05:40:57 UTC
pypolicyd-spf-1.3.2-4.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27

Comment 27 Fedora Update System 2017-01-18 04:19:08 UTC
pypolicyd-spf-2.0.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e

Comment 28 Fedora Update System 2017-01-18 04:19:23 UTC
pypolicyd-spf-2.0.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7

Comment 29 Fedora Update System 2017-01-19 07:23:14 UTC
pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7

Comment 30 Fedora Update System 2017-01-19 09:10:49 UTC
pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e

Comment 31 Fedora Update System 2017-01-29 00:20:05 UTC
pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 32 Fedora Update System 2017-01-29 00:49:13 UTC
pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.