Description of problem: found a buggy SPF record entry of a foreign domain in DNS causing policyd-spf crashing instead of simply ignoring that particular entry Version-Release number of selected component (if applicable): pypolicyd-spf-1.3.2-5.el7.noarch How reproducible: Always Steps to Reproduce (demo to reproduce created) 1. create test file: cat <<END /tmp/spfbuggy.txt request=smtpd_access_policy protocol_state=RCPT protocol_name=SMTP helo_name=mail.spfbuggy.demo.bieringer.de queue_id=1234567890 instance=12345.6789 sender=sender.bieringer.de recipient=test client_address=192.0.2.1 client_name=mail.spfbuggy.demo.bieringer.de END 2. execute manual check cat /tmp/spfbuggy.txt | /usr/libexec/postfix/policyd-spf ; echo $? Actual results: Traceback (most recent call last): File "/usr/libexec/postfix/policyd-spf", line 697, in <module> instance_dict, configData, peruser) File "/usr/libexec/postfix/policyd-spf", line 521, in _spfcheck res = spf.check2(ip, sender, helo, querytime=configData.get('Lookup_Time')) File "/usr/lib/python2.7/site-packages/spf.py", line 293, in check2 receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check() File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check rc = self.check1(spf, self.d, 0) File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1 return self.check0(spf, recursion) File "/usr/lib/python2.7/site-packages/spf.py", line 902, in check0 if self.cidrmatch([arg], cidrlength): break File "/usr/lib/python2.7/site-packages/spf.py", line 1355, in cidrmatch for netwrk in [ipaddress.IPNetwork(ip,strict=False) for ip in ipaddrs]: File "/usr/lib/python2.7/site-packages/ipaddr.py", line 118, in IPNetwork address) ValueError: '192.0.2.1\n' does not appear to be an IPv4 or IPv6 network 1 (this caused a lot of entry in maillog, had to whitelist particular client now) Expected results: being more relaxed and simply skip but log with level notice or warning the buggy entry Additional info: DNS entry: spfbuggy.demo.bieringer.de. 86400 IN TXT "v=spf1 a mx ip4:192.0.2.1\010 ip4:192.0.2.2 ~all" (have no clue how this was created, but taken from real example). btw: replacing "spfbuggy" with "spfok" will point you to a correct SPF record. https://mxtoolbox.com/SuperTool.aspx tells: + a Pass Match if IP has a DNS 'A' record in given domain + mx Pass Match if IP is one of the MX hosts for given domain name + error 192.0.2.1 Permerror Syntax Error near: 192.0.2.1 Is not a valid value for ip4 mechanism + ip4 192.0.2.2 Pass Match if IP is in the given range ~ all SoftFail Always matches. It goes at the end of your record. I haven't took a look what RFC tells about broken SPF record entries, but failing totally is not a good option imho....a try-catch block around particular record check would be the better one.
Participating files in the stack trace: rpm -qf /usr/lib/python2.7/site-packages/ipaddr.py python-ipaddr-2.1.11-1.el7.noarch rpm -qf /usr/lib/python2.7/site-packages/spf.py python-pyspf-2.0.11-5.el7.noarch rpm -qf /usr/libexec/postfix/policyd-spf pypolicyd-spf-1.3.2-5.el7.noarch
FEDORA-EPEL-2020-8d987efa26 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-8d987efa26
can't be reproduced anymore, also https://mxtoolbox.com/SuperTool.aspx is no longer claiming about...
able to reproduce it again, the piping is no longer working, one has to enter the SPF contents afterwards... /usr/libexec/postfix/policyd-spf request=smtpd_access_policy protocol_state=RCPT protocol_name=SMTP helo_name=mail.spfbuggy.demo.bieringer.de queue_id=1234567890 instance=12345.6789 sender=sender.bieringer.de recipient=test client_address=192.0.2.1 client_name=mail.spfbuggy.demo.bieringer.de Traceback (most recent call last): File "/usr/libexec/postfix/policyd-spf", line 697, in <module> instance_dict, configData, peruser) File "/usr/libexec/postfix/policyd-spf", line 521, in _spfcheck res = spf.check2(ip, sender, helo, querytime=configData.get('Lookup_Time')) File "/usr/lib/python2.7/site-packages/spf.py", line 293, in check2 receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check() File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check rc = self.check1(spf, self.d, 0) File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1 return self.check0(spf, recursion) File "/usr/lib/python2.7/site-packages/spf.py", line 902, in check0 if self.cidrmatch([arg], cidrlength): break File "/usr/lib/python2.7/site-packages/spf.py", line 1355, in cidrmatch for netwrk in [ipaddress.IPNetwork(ip,strict=False) for ip in ipaddrs]: File "/usr/lib/python2.7/site-packages/ipaddr.py", line 118, in IPNetwork address) ValueError: '192.0.2.1\n' does not appear to be an IPv4 or IPv6 network
Here more easy to execute tests: # /usr/bin/spfquery --ip 192.0.2.1 --sender sender.bieringer.de --helo mail.spfbuggy.demo.bieringer.de --rcpt-to=test Traceback (most recent call last): File "/usr/bin/spfquery", line 136, in <module> main(sys.argv[1:]) File "/usr/bin/spfquery", line 120, in main format(q) File "/usr/bin/spfquery", line 97, in format res,code,txt = q.check() File "/usr/lib/python2.7/site-packages/spf.py", line 543, in check rc = self.check1(spf, self.d, 0) File "/usr/lib/python2.7/site-packages/spf.py", line 582, in check1 return self.check0(spf, recursion) File "/usr/lib/python2.7/site-packages/spf.py", line 902, in check0 if self.cidrmatch([arg], cidrlength): break File "/usr/lib/python2.7/site-packages/spf.py", line 1355, in cidrmatch for netwrk in [ipaddress.IPNetwork(ip,strict=False) for ip in ipaddrs]: File "/usr/lib/python2.7/site-packages/ipaddr.py", line 118, in IPNetwork address) ValueError: '192.0.2.1\n' does not appear to be an IPv4 or IPv6 network # /usr/bin/spfquery --ip 192.0.2.1 --sender sender.bieringer.de --helo mail.spfok.demo.bieringer.de --rcpt-to=test pass () ('spfquery:', 'domain of spfok.demo.bieringer.de designates 192.0.2.1 as permitted sender') ('Received-SPF:', 'Pass (spfquery: domain of spfok.demo.bieringer.de designates 192.0.2.1 as permitted sender) client-ip=192.0.2.1; envelope-from="sender.bieringer.de"; helo=mail.spfok.demo.bieringer.de; receiver=spfquery; mechanism="ip4:192.0.2.1"; identity=mailfrom')
Following patch would help also for this semi-buggy SPF entries found @@ -587,6 +587,7 @@ if not spf: spf = self.dns_spf(self.d) if self.verbose: self.log("top",self.d,spf) + spf = spf.replace('\n','') if self.libspf_local and spf: spf = insert_libspf_local_policy( spf, self.libspf_local)
OK, I'll spin up -7 when I get some time and include your patch. Once again, thank you for testing!
python-pyspf-2.0.14-6.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-8d987efa26
(In reply to Peter Bieringer from comment #8) > Following patch would help also for this semi-buggy SPF entries found > > @@ -587,6 +587,7 @@ > if not spf: > spf = self.dns_spf(self.d) > if self.verbose: self.log("top",self.d,spf) > + spf = spf.replace('\n','') > if self.libspf_local and spf: > spf = insert_libspf_local_policy( > spf, self.libspf_local) Just spent some time reading SPF RFC and I am not sure whether we are going to be breaking SPF format if this is done. I know a lot of people probably mis-configured their TXT/SPF records, but I'm worried we may introduce a non-standard behaviour with this. Here is a question, does the package as is work with well formed TXT/SPF records?
sorry, the patch shown above was the first one I've tried, correct one should replace newline by a space (and not simply removing it) @@ -587,6 +587,7 @@ if not spf: spf = self.dns_spf(self.d) if self.verbose: self.log("top",self.d,spf) + spf = spf.replace('\n',' ') if self.libspf_local and spf: spf = insert_libspf_local_policy( spf, self.libspf_local) Regarding testing of a broken SPF record placed in spfbuggy.demo.bieringer.de https://mxtoolbox.com/SuperTool.aspx -> OK https://www.dmarcanalyzer.com/de/spf-de/checker/ -> OK https://www.spf-record.de/spf-lookup -> OK => all 3 are not claiming about the newline in the record https://www.kitterman.com/spf/validate.html => reporting an issue, but using this particular library...so no wonder
OK, thanks for providing additional info. I guess the idea is to relax a bit about newline mid-record and just turn it into a space, so that incorrectly coded SPF/TXT records can still pass validation. I know this is not 100% RFC compliant, but it seems to be practical and I cannot see a downside to it. I'll integrate your patch into the next builds.
FEDORA-2020-9e8c8dedb3 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-9e8c8dedb3
FEDORA-EPEL-2020-08c57e33f8 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-08c57e33f8
FEDORA-EPEL-2020-582b05fe3c has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-582b05fe3c
FEDORA-2020-3fa8f469a0 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2020-3fa8f469a0
python-pyspf-2.0.14-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3fa8f469a0
python-pyspf-2.0.14-2.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-9e8c8dedb3
python-pyspf-2.0.14-2.el8 has been pushed to the Fedora EPEL 8 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-582b05fe3c
python-pyspf-2.0.14-7.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-08c57e33f8
python-pyspf-2.0.14-8.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-08c57e33f8
python-pyspf-2.0.14-9.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-08c57e33f8
python-pyspf-2.0.14-3.el8 has been pushed to the Fedora EPEL 8 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-582b05fe3c
python-pyspf-2.0.14-3.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3fa8f469a0
python-pyspf-2.0.14-3.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-9e8c8dedb3
python-pyspf-2.0.14-10.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-08c57e33f8
python-pyspf-2.0.14-3.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.
python-pyspf-2.0.14-3.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
python-pyspf-2.0.14-3.el8 has been pushed to the Fedora EPEL 8 stable repository. If problems still persist, please make note of it in this bug report.
python-pyspf-2.0.14-10.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
(In reply to Peter Bieringer from comment #12) > sorry, the patch shown above was the first one I've tried, correct one > should replace newline by a space (and not simply removing it) > > > @@ -587,6 +587,7 @@ > if not spf: > spf = self.dns_spf(self.d) > if self.verbose: self.log("top",self.d,spf) > + spf = spf.replace('\n',' ') > if self.libspf_local and spf: > spf = insert_libspf_local_policy( > spf, self.libspf_local) > > Regarding testing of a broken SPF record placed in spfbuggy.demo.bieringer.de > > https://mxtoolbox.com/SuperTool.aspx -> OK > > https://www.dmarcanalyzer.com/de/spf-de/checker/ -> OK > > https://www.spf-record.de/spf-lookup -> OK > https://emailauth.io/dmarc-generator -> OK > => all 3 are not claiming about the newline in the record > > > https://www.kitterman.com/spf/validate.html > > => reporting an issue, but using this particular library...so no wonder