Description of problem: Some emails sent through the libreoffice mailing list cause policyd-spf to issue a traceback causing emails to be rejected with 451 Configuation error. I am currently only seeing this with SOME emails from libreoffice mailing list and so far no where else. Version-Release number of selected component (if applicable): pypolicyd-spf-1.3.1-3.fc23.noarch How reproducible: Not sure. Not all through the list do it. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: Dec 5 07:05:01 SystemA postfix/postscreen[3995]: CONNECT from [2a01:4f8:190:3144::2]:40800 to [2001:470:1f11:1007::1]:25 Dec 5 07:05:01 SystemA postfix/postscreen[3995]: PASS OLD [2a01:4f8:190:3144::2]:40800 Dec 5 07:05:01 SystemA postfix/smtpd[3996]: connect from bilbo2.documentfoundation.org[2a01:4f8:190:3144::2] Dec 5 07:05:02 SystemA postfix/smtpd[3996]: Anonymous TLS connection established from bilbo2.documentfoundation.org[2a01:4f8:190:3144::2]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) Dec 5 07:05:03 SystemA policyd-spf[4002]: Traceback (most recent call last): Dec 5 07:05:03 SystemA policyd-spf[4002]: File "/usr/libexec/postfix/policyd-spf", line 687, in <module> Dec 5 07:05:03 SystemA policyd-spf[4002]: instance_dict, configData, peruser) Dec 5 07:05:03 SystemA policyd-spf[4002]: File "/usr/libexec/postfix/policyd-spf", line 420, in _spfcheck Dec 5 07:05:03 SystemA policyd-spf[4002]: res = spf.check2(ip, helo_fake_sender, helo, querytime=configData.get('Lookup_Time')) Dec 5 07:05:03 SystemA policyd-spf[4002]: File "/usr/lib/python2.7/site-packages/spf.py", line 290, in check2 Dec 5 07:05:03 SystemA policyd-spf[4002]: receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check() Dec 5 07:05:03 SystemA policyd-spf[4002]: File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check Dec 5 07:05:03 SystemA policyd-spf[4002]: rc = self.check1(spf, self.d, 0) Dec 5 07:05:03 SystemA policyd-spf[4002]: File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1 Dec 5 07:05:03 SystemA policyd-spf[4002]: return self.check0(spf, recursion) Dec 5 07:05:03 SystemA policyd-spf[4002]: File "/usr/lib/python2.7/site-packages/spf.py", line 891, in check0 Dec 5 07:05:03 SystemA policyd-spf[4002]: if self.cidrmatch(self.dns_a(arg,self.A), cidrlength): Dec 5 07:05:03 SystemA policyd-spf[4002]: File "/usr/lib/python2.7/site-packages/spf.py", line 1202, in dns_a Dec 5 07:05:03 SystemA policyd-spf[4002]: return [ipaddress.Bytes(ip) for ip in r] Dec 5 07:05:03 SystemA policyd-spf[4002]: AttributeError: 'module' object has no attribute 'Bytes' Dec 5 07:05:03 SystemA postfix/spawn[4001]: warning: command /usr/libexec/postfix/policyd-spf exit status 1 Dec 5 07:05:03 SystemA postfix/smtpd[3996]: warning: premature end-of-input on private/policy-spf while reading input attribute name Dec 5 07:05:04 SystemA policyd-spf[4004]: Traceback (most recent call last): Dec 5 07:05:04 SystemA policyd-spf[4004]: File "/usr/libexec/postfix/policyd-spf", line 687, in <module> Dec 5 07:05:04 SystemA policyd-spf[4004]: instance_dict, configData, peruser) Dec 5 07:05:04 SystemA policyd-spf[4004]: File "/usr/libexec/postfix/policyd-spf", line 420, in _spfcheck Dec 5 07:05:04 SystemA policyd-spf[4004]: res = spf.check2(ip, helo_fake_sender, helo, querytime=configData.get('Lookup_Time')) Dec 5 07:05:04 SystemA policyd-spf[4004]: File "/usr/lib/python2.7/site-packages/spf.py", line 290, in check2 Dec 5 07:05:04 SystemA policyd-spf[4004]: receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check() Dec 5 07:05:04 SystemA policyd-spf[4004]: File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check Dec 5 07:05:04 SystemA policyd-spf[4004]: rc = self.check1(spf, self.d, 0) Dec 5 07:05:04 SystemA policyd-spf[4004]: File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1 Dec 5 07:05:04 SystemA policyd-spf[4004]: return self.check0(spf, recursion) Dec 5 07:05:04 SystemA policyd-spf[4004]: File "/usr/lib/python2.7/site-packages/spf.py", line 891, in check0 Dec 5 07:05:04 SystemA policyd-spf[4004]: if self.cidrmatch(self.dns_a(arg,self.A), cidrlength): Dec 5 07:05:04 SystemA policyd-spf[4004]: File "/usr/lib/python2.7/site-packages/spf.py", line 1202, in dns_a Dec 5 07:05:04 SystemA policyd-spf[4004]: return [ipaddress.Bytes(ip) for ip in r] Dec 5 07:05:04 SystemA policyd-spf[4004]: AttributeError: 'module' object has no attribute 'Bytes' Dec 5 07:05:04 SystemA postfix/spawn[4001]: warning: command /usr/libexec/postfix/policyd-spf exit status 1 Dec 5 07:05:04 SystemA postfix/smtpd[3996]: warning: premature end-of-input on private/policy-spf while reading input attribute name Dec 5 07:05:04 SystemA postfix/smtpd[3996]: warning: problem talking to server private/policy-spf: Success I imagine some of the code here needs to be a in a try block or some other error prevention/catching method.
The problem is that with a switch to python-ipaddress instead of python-ipaddr, some stuff appears to be missing from the API. So, python-pyspf will need to be patched again to fix this.
Total guess here (I'm not a python guy at all), but does this patch do anything for you? ---------------------- diff -ruN pyspf-2.0.11-v/spf.py pyspf-2.0.11/spf.py --- pyspf-2.0.11-v/spf.py 2014-12-06 03:20:07.000000000 +1100 +++ pyspf-2.0.11/spf.py 2015-12-08 11:41:35.025136287 +1100 @@ -1203,7 +1203,7 @@ 'No %s records found for'%A, domainname) if A == 'AAAA' and bytes is str: # work around pydns inconsistency plus python2 bytes/str ambiguity - return [ipaddress.Bytes(ip) for ip in r] + return [Bytes(ip) for ip in r] return r def validated_ptrs(self): @@ -1617,6 +1617,22 @@ % (sender, self.c) raise ValueError("invalid SPF result for header comment: "+res) +# We need to distinguish between the string and packed-bytes representations +# of an IP address. For example, b'0::1' is the IPv4 address 48.58.58.49, +# while '0::1' is an IPv6 address. +# +# In Python 3, the native 'bytes' type already provides this functionality, +# so we use it directly. For earlier implementations where bytes is not a +# distinct type, we create a subclass of str to serve as a tag. +try: + if bytes is str: + raise TypeError("bytes is not a distinct type") + Bytes = bytes +except (NameError, TypeError): + class Bytes(str): + def __repr__(self): + return 'Bytes(%s)' % str.__repr__(self) + def split_email(s, h): """Given a sender email s and a HELO domain h, create a valid tuple (l, d) local-part and domain-part. ----------------------
This patch seems to break all IPv6 connections as I am not getting the error between two domains that have the same configuration (I do not believe it was there before) and with ack.nmap.org.
(In reply to Trever Adams from comment #3) > This patch seems to break all IPv6 connections as I am not getting the error > between two domains that have the same configuration (I do not believe it > was there before) and with ack.nmap.org. Thanks for testing and sorry about the failure. I have absolutely no idea what I'm doing there. I think someone with the knowledge of python and the code in python-pyspf will need to have a look.
With the patch the traceback changes. Dec 9 15:13:53 SystemA policyd-spf[25731]: None; identity=helo; client-ip=2600:3c01::f03c:91ff:fe98:ff4e; helo=ack.nmap.org; envelope-from=fulldisclosure-bounces; receiver=trever.org Dec 9 15:13:53 SystemA policyd-spf[25731]: Traceback (most recent call last): Dec 9 15:13:53 SystemA policyd-spf[25731]: File "/usr/libexec/postfix/policyd-spf", line 687, in <module> Dec 9 15:13:53 SystemA policyd-spf[25731]: instance_dict, configData, peruser) Dec 9 15:13:53 SystemA policyd-spf[25731]: File "/usr/libexec/postfix/policyd-spf", line 524, in _spfcheck Dec 9 15:13:53 SystemA policyd-spf[25731]: res = spf.check2(ip, sender, helo, querytime=configData.get('Lookup_Time')) Dec 9 15:13:53 SystemA policyd-spf[25731]: File "/usr/lib/python2.7/site-packages/spf.py", line 290, in check2 Dec 9 15:13:53 SystemA policyd-spf[25731]: receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check() Dec 9 15:13:53 SystemA policyd-spf[25731]: File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check Dec 9 15:13:53 SystemA policyd-spf[25731]: rc = self.check1(spf, self.d, 0) Dec 9 15:13:53 SystemA policyd-spf[25731]: File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1 Dec 9 15:13:53 SystemA policyd-spf[25731]: return self.check0(spf, recursion) Dec 9 15:13:53 SystemA policyd-spf[25731]: File "/usr/lib/python2.7/site-packages/spf.py", line 891, in check0 Dec 9 15:13:53 SystemA policyd-spf[25731]: if self.cidrmatch(self.dns_a(arg,self.A), cidrlength): Dec 9 15:13:53 SystemA policyd-spf[25731]: File "/usr/lib/python2.7/site-packages/spf.py", line 1347, in cidrmatch Dec 9 15:13:53 SystemA policyd-spf[25731]: ipaddrs[idx] = ipaddrs[idx].decode('ascii') Dec 9 15:13:53 SystemA policyd-spf[25731]: UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 8: ordinal not in range(128) Dec 9 15:13:53 SystemA postfix/spawn[25191]: warning: command /usr/libexec/postfix/policyd-spf exit status 1 Dec 9 15:13:53 SystemA postfix/smtpd[25551]: warning: premature end-of-input on private/policy-spf while reading input attribute name Dec 9 15:13:53 SystemA postfix/smtpd[25551]: warning: problem talking to server private/policy-spf: Success Dec 9 15:13:53 SystemA postfix/smtpd[25551]: NOQUEUE: reject: RCPT from ack.nmap.org[2600:3c01::f03c:91ff:fe98:ff4e]: 451 4.3.5 <trever.org>: Recipient address rejected: Server configuration problem; from=<fulldisclosure-bounces> to=<trever.org> proto=ESMTP helo=<ack.nmap.org>
Oh, and it is still not 100% consistent. Is it possible this is being caused by dns lookups being a bit slow at times? (My CenturyLink pppoe connection is not the best and stalls at times due to them refusing to fix some problems with the line.)
OK, one last try before I give up: ------------------- diff -ruN pyspf-2.0.11-v/spf.py pyspf-2.0.11/spf.py --- pyspf-2.0.11-v/spf.py 2014-12-06 03:20:07.000000000 +1100 +++ pyspf-2.0.11/spf.py 2015-12-11 13:31:46.102844734 +1100 @@ -1201,9 +1201,6 @@ if self.strict > 1 and len(r) == 0: raise AmbiguityWarning( 'No %s records found for'%A, domainname) - if A == 'AAAA' and bytes is str: - # work around pydns inconsistency plus python2 bytes/str ambiguity - return [ipaddress.Bytes(ip) for ip in r] return r def validated_ptrs(self): ------------------- Essentially, don't touch what's returned, but just pass it on. Absolutely no idea whether this will screw things up even more.
Is this in lieu of or in addition to the other patch?
Just on its own.
It does not solve the problem. Whatever it is is transient. Messages get rejected and then accepted later. I have no clue what is going on other than the traceback. IPv6 seems to be the only commonality.
I'm going to assign this to python-ipaddress package now, so that folks maintaining that can point us in the right direction. I have no idea how this is supposed to work, what should play the role of ipaddress.Bytes here etc.
This problem still exists in fc24 with the following packages. python-ipaddress-1.0.16-2.fc24.noarch python-pyspf-2.0.11-4.fc24.noarch
Unfortunately, my DSL provider is a bit flaky at times. 90% of the problems happen within 1 minute of the ppp connection resetting. The rest of the time, I am assuming the connection has slowed enough to cause a timeout of the dns request. So, this problem isn't a bug in the normal code path, but either a bug in an error path or the lack thereof.
After another month of watching, yes, the connection dropping and therefore DNS problems will cause this, but that isn't it. the address listed in the title of this report yields IPv6 results. It happens very consistently. If it comes in over IPv4 there is no problem.
It appears that it happens far more commonly with IPv6 addresses with ::.
I've just built this in F26: http://koji.fedoraproject.org/koji/buildinfo?buildID=820968 I haven't tested it at all, but it is python3 based. Maybe that will be better with IPv6 addresses. No idea. Anyhow, there it is, in case you want to try it.
I am not seeing the required python3-pyspf. Is that built somewhere?
My mistake, koji only lists it under python-pyspf.
This may have fixed it. I got a bunch from the documentfoundation mailing list today with no problems. I will give it a few more days before I say it is fixed. If it is, can we see this in F25?
(In reply to Trever Adams from comment #19) > If it is, can we see this in F25? Absolutely. And F24, if all goes as planned with pyspf maintainers.
pypolicyd-spf-1.3.2-4.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c
pypolicyd-spf-1.3.2-4.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27
Thank you Bojan, I can comfortably confirm that the bug has been fixed by moving to pure python3. Thank you! Please, feel free to close this bug as fixed with the packages in question when all parties have pushed their packages. Thank you all!
(In reply to Trever Adams from comment #23) > Thank you Bojan, I can comfortably confirm that the bug has been fixed by > moving to pure python3. Thank you! Please, feel free to close this bug as > fixed with the packages in question when all parties have pushed their > packages. > > Thank you all! Thank you for testing. The bug will be automatically closed when the updated packages reach stable.
pypolicyd-spf-1.3.2-4.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c
pypolicyd-spf-1.3.2-4.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27
pypolicyd-spf-2.0.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e
pypolicyd-spf-2.0.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7
pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7
pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e
pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.