Bug 1374925

Inertia. Lack of dependencies in python3 in the past. Laziness of the maintainer. Take your pick. :-)

I just had a quick peek at this today. Essentially, there is package python-pyspf in Fedora, which is a python2 package. So, unless we get that built for python3, we won't have this package running under python3.

Please see:

http://koji.fedoraproject.org/koji/buildinfo?buildID=820968

Totally untested, so any feedback is very welcome.

i would need an F24 scratch-build to install it on my testserver

Unfortunately, python3-pyspf is only available for F26. So, probably the "easiest" thing to do is to grab that from F26 (still built as python-pyspf, but contains both python2 and python3 stuff) and try to install alongside pypolicyd-spf from F26.

Not quite sure whether that will try to pull in a whole lot of other stuff, but it's worth a try.

Cannot speak for F24, because I have none lying around any more, but this is what happens on F25:

# dnf install https://kojipkgs.fedoraproject.org//packages/pypolicyd-spf/1.3.2/4.fc26/noarch/pypolicyd-spf-1.3.2-4.fc26.noarch.rpm https://kojipkgs.fedoraproject.org//packages/python-pyspf/2.0.12/1.fc26/noarch/python3-pyspf-2.0.12-1.fc26.noarch.rpm
Last metadata expiration check: 2:54:31 ago on Tue Nov 29 09:52:15 2016.
Dependencies resolved.
================================================================================
 Package              Arch         Version             Repository          Size
================================================================================
Installing:
 python3-py3dns       noarch       3.0.4-8.fc25        fedora              47 k
 python3-pyspf        noarch       2.0.12-1.fc26       @commandline        52 k
Upgrading:
 pypolicyd-spf        noarch       1.3.2-4.fc26        @commandline        48 k

Transaction Summary
================================================================================
Install  2 Packages
Upgrade  1 Package

Total size: 148 k
Total download size: 47 k
Is this ok [y/N]:

Had to also add:

https://kojipkgs.fedoraproject.org//packages/python-pyspf/2.0.12/1.fc26/noarch/python2-pyspf-2.0.12-1.fc26.noarch.rpm

To that dnf install. This replaces existing F25 python-pyspf, in order to avoid conflicts.

I don't have F24 lying around to test F26 builds. I tested them on F25 and they work.

Have you tried them on F24?

If they work for you, then we can ask the maintainer of pyspf to backport and do everything properly.

I know how to build scratch builds.

What is interesting is the fact /usr/bin/python is dying. Not sure how that's happening when F26 package executable has /usr/bin/python3 in the shebang line.

I can make you a scratch build for F24, but unless you also rebuild pyspf for F24, you'll still have to use that F26 build for testing.

we need srcatch build for the depending packages too - it makes just no sense to play around on F24 machines in context of a F24 bugreport with F26 packages from the start

I don't maintain the other package. I think your best bet is to rebuild both locally and give it a try.

I am planning on backporting this to F24/25, but pyspf maintainer will have to rebuild that first (not hard).

Not everything was ported to python3 in F23. If I remember correctly, we were still mucking about with ipaddr/up address stuff at that point, maybe even later.

(In reply to Harald Reindl from comment #10)

> Nov 29 11:54:31 testserver postfix/spawn[18157]: warning: command
> /usr/bin/python exit status 1

Can I just ask regarding this, what is in your /etc/postfix/master.cf file when it comes to policyd-spf? The first line of /usr/libexec/postfix/policyd-spf is:

#!/usr/bin/python3 -s

No idea why /usr/bin/python would be invoked on your machine. On my F25 machine, that is a symlink to python2.

why is this still a symlink to python2
that's not really portable in case of "Python_3_as_Default"

[root@testserver:~]$ cat /etc/postfix/master.cf | grep spf
spf-policy-info unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf-noreject.conf
spf-policy      unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf.conf

(In reply to Harald Reindl from comment #17)
> why is this still a symlink to python2
> that's not really portable in case of "Python_3_as_Default"

Cannot really answer that, to be honest. I don't make such decisions.

> [root@testserver:~]$ cat /etc/postfix/master.cf | grep spf
> spf-policy-info unix  -       n       n       -        0      spawn
> user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf
> /etc/python-policyd-spf/policyd-spf-noreject.conf
> spf-policy      unix  -       n       n       -        0      spawn
> user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf
> /etc/python-policyd-spf/policyd-spf.conf

Try removing /usr/bin/python from argv and running that again. Python3 should then be picked up by the script and so should the relevant pyspf python3 stuff.

pypolicyd-spf-1.3.2-4.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c

pypolicyd-spf-1.3.2-4.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27

pypolicyd-spf-1.3.2-4.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c

pypolicyd-spf-1.3.2-4.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27

looks good and the also "python2-pyspf" can be removed afterwards which is important to be able uninstall finally python2, postfix config needs to explizit use "python3" since /usr/bin/python is stll python2 and on machines where python2 is uninstalled there is no /usr/bin/python at all but python3
___________________________________________________

[root@testserver:~]$ cat /etc/postfix/master.cf | grep spf
spf-policy-info unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python3 /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf-noreject.conf
spf-policy      unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python3 /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf.conf
___________________________________________________

[root@testserver:~]$ rpm -qa | grep -i spf
perl-Mail-SPF-2.9.0-8.fc24.noarch
pypolicyd-spf-1.3.2-4.fc24.noarch
python3-pyspf-2.0.12-1.fc24.noarch
___________________________________________________

Dec  3 13:18:10 testserver postfix/smtpd[30101]: connect from local.rhsoft.net[62.178.103.85]
Dec  3 13:18:12 testserver policyd-spf[30109]: Pass; identity=helo; client-ip=62.178.103.85; helo=srv-rhsoft.rhsoft.net; envelope-from=h.reindl; receiver=rhsoft
Dec  3 13:18:12 testserver policyd-spf[30109]: Fail; identity=mailfrom; client-ip=62.178.103.85; helo=srv-rhsoft.rhsoft.net; envelope-from=h.reindl; receiver=rhsoft
Dec  3 13:18:12 testserver postfix/smtpd[30101]: NOQUEUE: reject: RCPT from local.rhsoft.net[62.178.103.85]: 550 5.7.1 <rhsoft>: Recipient address rejected: Message rejected due to: SPF fail - not authorized. Please see http://www.openspf.net/Why?s=mfrom;id=h.reindl@thelounge.net;ip=62.178.103.85;r=rhsoft@test.rh; from=<h.reindl> to=<rhsoft> proto=ESMTP helo=<srv-rhsoft.rhsoft.net>
Dec  3 13:18:12 testserver postfix/smtpd[30101]: lost connection after RCPT from local.rhsoft.net[62.178.103.85]
Dec  3 13:18:12 testserver postfix/smtpd[30101]: disconnect from local.rhsoft.net[62.178.103.85] ehlo=1 mail=1 rcpt=0/1 commands=2/3

i guess that only becomes more visible with Python3

"OSError: [Errno 98] Address already in use" - why does that stuff deal at it's own with the source-port? on a busy machine running dns-resolvers and all sort of milters and deamons for mail-processing that is racy by definition

Dec  6 12:03:20 mail-gw policyd-spf[8410]: Traceback (most recent call last):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 198, in getSource
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.s.bind(('', source_port))
Dec  6 12:03:20 mail-gw policyd-spf[8410]: OSError: [Errno 98] Address already in use
Dec  6 12:03:20 mail-gw policyd-spf[8410]: 
Dec  6 12:03:20 mail-gw policyd-spf[8410]: During handling of the above exception, another exception occurred:
Dec  6 12:03:20 mail-gw policyd-spf[8410]: Traceback (most recent call last):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/libexec/postfix/policyd-spf", line 707, in <module>
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    instance_dict, configData, peruser)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/libexec/postfix/policyd-spf", line 419, in _spfcheck
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    res = spf.check2(ip, helo_fake_sender, helo, querytime=configData.get('Lookup_Time'))
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 304, in check2
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 569, in check
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    rc = self.check1(spf, self.d, 0)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 608, in check1
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    return self.check0(spf, recursion)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 917, in check0
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    if self.cidrmatch(self.dns_a(arg,self.A), cidrlength):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 1222, in dns_a
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    r = self.dns(domainname, A)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 1310, in dns
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    for k, v in DNSLookup(name, qtype, self.strict, timeout):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 127, in DNSLookup
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    resp = req.req()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 244, in req
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.sendUDPRequest(server)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 269, in sendUDPRequest
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.conn()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 205, in conn
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.getSource()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 202, in getSource
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    if msg[0] != 98: raise
Dec  6 12:03:20 mail-gw policyd-spf[8410]: TypeError: 'OSError' object is not subscriptable

(In reply to Harald Reindl from comment #24)
> i guess that only becomes more visible with Python3
> 
> "OSError: [Errno 98] Address already in use" - why does that stuff deal at
> it's own with the source-port? on a busy machine running dns-resolvers and
> all sort of milters and deamons for mail-processing that is racy by
> definition

I think you should open an upstream bug for this (or at least another one in Red Hat Bugzilla). I haven't personally seen this, but then my config may be different.

https://bugzilla.redhat.com/show_bug.cgi?id=1408273

pypolicyd-spf-2.0.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e

pypolicyd-spf-2.0.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7

pypolicyd-spf-2.0.1-1.fc24 has a regression 

look at the receiver=<UNKNOWN> that is now on every logline and header besides that the logformat slightly changed

Jan 18 11:38:25 mail-gw policyd-spf[27152]: prepend Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=66.220.144.147; helo=mx-out.facebook.com; envelope-from=****; receiver=<UNKNOWN>

Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=62.210.161.103;
 helo=r103.p3.neolane.net; envelope-from****;
 receiver=<UNKNOWN>

(In reply to Harald Reindl from comment #29)
> pypolicyd-spf-2.0.1-1.fc24 has a regression 
> 
> look at the receiver=<UNKNOWN> that is now on every logline and header
> besides that the logformat slightly changed
> 
> Jan 18 11:38:25 mail-gw policyd-spf[27152]: prepend Received-SPF: Pass
> (mailfrom) identity=mailfrom; client-ip=66.220.144.147;
> helo=mx-out.facebook.com; envelope-from=****; receiver=<UNKNOWN>
> 
> Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=62.210.161.103;
>  helo=r103.p3.neolane.net; envelope-from****;
>  receiver=<UNKNOWN>

From the documentation of 2.0.1:

# In order to avoid disclosing BCC recipients in SPF header fields,
# Hide_Receiver is set to Yes by default in the interest of maximizing
# privacy.  This setting will replace the actual recipient with <UNKNOWN> both
# in header fields and SMTP responses.  The latter may make it more difficult
# for senders to troubleshoot issues with their SPF deployments.
#Hide_Receiver = No
Hide_Receiver = Yes

thanks - i turn my karma to +1 with that option

pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7

pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e

BTW: can someone ping the "logwatch" maintainers - the unnecessary change of logging to "prepend Received-SPF: " leads to blow all entries into **Unmatched Entries** in the daily logwatch mails - i would really love when developers stop fix things which ain't broken

(In reply to Harald Reindl from comment #34)
> BTW: can someone ping the "logwatch" maintainers

Probably best if you open a logwatch bug in Fedora. Maintainers will then take it from there.

pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.