Bug 1374925 - why is pypolicyd-spf on F24 still using python2
Summary: why is pypolicyd-spf on F24 still using python2
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: pypolicyd-spf
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Bojan Smojver
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-10 12:54 UTC by Harald Reindl
Modified: 2019-01-07 14:37 UTC (History)
3 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2017-01-29 00:20:10 UTC


Attachments (Terms of Use)

Description Harald Reindl 2016-09-10 12:54:00 UTC
upstream supports python3 which is the default in Fedora 24 for over 4 years now but F24 is stlll using python2 leading you can't get rid off python2 and have the whole python3 stack additional on small machines becasue of system deps like dnf


https://launchpad.net/pypolicyd-spf/+announcement/10352
pypolicyd-spf now supports Python 3

Written for pypolicyd-spf by Scott Kitterman on 2012-07-22

As of version 1.1, released today, pypolicyd-spf can be run with any of python2.6, python2.7, and python3.2.

Version 1.1 is functionally identical to version 1.0, so there is no need to upgrade unless you prefer to use Python 3. Users of older Python versions (python2.4 and python2.5 will need to stay with version 1.0).

Comment 1 Bojan Smojver 2016-09-11 04:48:56 UTC
Inertia. Lack of dependencies in python3 in the past. Laziness of the maintainer. Take your pick. :-)

Comment 2 Bojan Smojver 2016-10-25 05:53:14 UTC
I just had a quick peek at this today. Essentially, there is package python-pyspf in Fedora, which is a python2 package. So, unless we get that built for python3, we won't have this package running under python3.

Comment 3 Bojan Smojver 2016-11-29 00:34:48 UTC
Please see:

http://koji.fedoraproject.org/koji/buildinfo?buildID=820968

Totally untested, so any feedback is very welcome.

Comment 4 Harald Reindl 2016-11-29 00:54:22 UTC
i would need an F24 scratch-build to install it on my testserver

Comment 5 Bojan Smojver 2016-11-29 01:41:10 UTC
Unfortunately, python3-pyspf is only available for F26. So, probably the "easiest" thing to do is to grab that from F26 (still built as python-pyspf, but contains both python2 and python3 stuff) and try to install alongside pypolicyd-spf from F26.

Not quite sure whether that will try to pull in a whole lot of other stuff, but it's worth a try.

Comment 6 Bojan Smojver 2016-11-29 01:48:01 UTC
Cannot speak for F24, because I have none lying around any more, but this is what happens on F25:

# dnf install https://kojipkgs.fedoraproject.org//packages/pypolicyd-spf/1.3.2/4.fc26/noarch/pypolicyd-spf-1.3.2-4.fc26.noarch.rpm https://kojipkgs.fedoraproject.org//packages/python-pyspf/2.0.12/1.fc26/noarch/python3-pyspf-2.0.12-1.fc26.noarch.rpm
Last metadata expiration check: 2:54:31 ago on Tue Nov 29 09:52:15 2016.
Dependencies resolved.
================================================================================
 Package              Arch         Version             Repository          Size
================================================================================
Installing:
 python3-py3dns       noarch       3.0.4-8.fc25        fedora              47 k
 python3-pyspf        noarch       2.0.12-1.fc26       @commandline        52 k
Upgrading:
 pypolicyd-spf        noarch       1.3.2-4.fc26        @commandline        48 k

Transaction Summary
================================================================================
Install  2 Packages
Upgrade  1 Package

Total size: 148 k
Total download size: 47 k
Is this ok [y/N]:

Comment 7 Bojan Smojver 2016-11-29 02:07:25 UTC
Had to also add:

https://kojipkgs.fedoraproject.org//packages/python-pyspf/2.0.12/1.fc26/noarch/python2-pyspf-2.0.12-1.fc26.noarch.rpm

To that dnf install. This replaces existing F25 python-pyspf, in order to avoid conflicts.

Comment 8 Harald Reindl 2016-11-29 07:42:47 UTC
how does it matter if you have F24 lying around for a https://fedoraproject.org/wiki/Using_the_Koji_build_system#Scratch_Builds while it matters that i as reporter hd a good reason to choose F24 for that bugreport and have no F25/F26 lying around which won't change in the next few months and my intention is to red rid of Python2 on our inbound mailserver aka production machine while on my testmachines it's neeeded anyways because of tue un-usability of DNF in borercases

Comment 9 Bojan Smojver 2016-11-29 08:00:13 UTC
I don't have F24 lying around to test F26 builds. I tested them on F25 and they work.

Have you tried them on F24?

If they work for you, then we can ask the maintainer of pyspf to backport and do everything properly.

Comment 10 Harald Reindl 2016-11-29 10:56:16 UTC
you can install the packages with "dnf install" since dnf update "still don't work" (https://bugzilla.redhat.com/show_bug.cgi?id=1263888#c63) but as explected F26 builds don't run well at F24

*please* read https://fedoraproject.org/wiki/Using_the_Koji_build_system#Scratch_Builds

Nov 29 11:54:31 testserver postfix/spawn[18157]: warning: command /usr/bin/python exit status 1
Nov 29 11:54:31 testserver postfix/smtpd[18151]: warning: premature end-of-input on private/spf-policy-info while reading input attribute name
Nov 29 11:54:32 testserver postfix/spawn[18157]: warning: command /usr/bin/python exit status 1
Nov 29 11:54:32 testserver postfix/smtpd[18151]: warning: premature end-of-input on private/spf-policy-info while reading input attribute name
Nov 29 11:54:32 testserver postfix/smtpd[18151]: warning: problem talking to server private/spf-policy-info: Connection reset by peer

Comment 11 Bojan Smojver 2016-11-29 11:10:27 UTC
I know how to build scratch builds.

What is interesting is the fact /usr/bin/python is dying. Not sure how that's happening when F26 package executable has /usr/bin/python3 in the shebang line.

I can make you a scratch build for F24, but unless you also rebuild pyspf for F24, you'll still have to use that F26 build for testing.

Comment 12 Harald Reindl 2016-11-29 11:12:42 UTC
we need srcatch build for the depending packages too - it makes just no sense to play around on F24 machines in context of a F24 bugreport with F26 packages from the start

Comment 13 Bojan Smojver 2016-11-29 11:25:32 UTC
I don't maintain the other package. I think your best bet is to rebuild both locally and give it a try.

Comment 14 Harald Reindl 2016-11-29 11:41:36 UTC
sorry but then i have to wait another two years while that changes should have happened for Fedora 23 and not Fedora 26

https://fedoraproject.org/wiki/Changes/Python_3_as_Default

Comment 15 Bojan Smojver 2016-11-29 12:04:32 UTC
I am planning on backporting this to F24/25, but pyspf maintainer will have to rebuild that first (not hard).

Not everything was ported to python3 in F23. If I remember correctly, we were still mucking about with ipaddr/up address stuff at that point, maybe even later.

Comment 16 Bojan Smojver 2016-11-30 00:25:39 UTC
(In reply to Harald Reindl from comment #10)

> Nov 29 11:54:31 testserver postfix/spawn[18157]: warning: command
> /usr/bin/python exit status 1

Can I just ask regarding this, what is in your /etc/postfix/master.cf file when it comes to policyd-spf? The first line of /usr/libexec/postfix/policyd-spf is:

#!/usr/bin/python3 -s

No idea why /usr/bin/python would be invoked on your machine. On my F25 machine, that is a symlink to python2.

Comment 17 Harald Reindl 2016-11-30 00:28:30 UTC
why is this still a symlink to python2
that's not really portable in case of "Python_3_as_Default"

[root@testserver:~]$ cat /etc/postfix/master.cf | grep spf
spf-policy-info unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf-noreject.conf
spf-policy      unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf.conf

Comment 18 Bojan Smojver 2016-11-30 00:56:28 UTC
(In reply to Harald Reindl from comment #17)
> why is this still a symlink to python2
> that's not really portable in case of "Python_3_as_Default"

Cannot really answer that, to be honest. I don't make such decisions.

> [root@testserver:~]$ cat /etc/postfix/master.cf | grep spf
> spf-policy-info unix  -       n       n       -        0      spawn
> user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf
> /etc/python-policyd-spf/policyd-spf-noreject.conf
> spf-policy      unix  -       n       n       -        0      spawn
> user=nobody argv=/usr/bin/python /usr/libexec/postfix/policyd-spf
> /etc/python-policyd-spf/policyd-spf.conf

Try removing /usr/bin/python from argv and running that again. Python3 should then be picked up by the script and so should the relevant pyspf python3 stuff.

Comment 19 Fedora Update System 2016-12-01 03:17:55 UTC
pypolicyd-spf-1.3.2-4.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c

Comment 20 Fedora Update System 2016-12-01 03:18:05 UTC
pypolicyd-spf-1.3.2-4.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27

Comment 21 Fedora Update System 2016-12-03 04:34:26 UTC
pypolicyd-spf-1.3.2-4.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-12ccd1d81c

Comment 22 Fedora Update System 2016-12-03 05:41:02 UTC
pypolicyd-spf-1.3.2-4.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-c4082feb27

Comment 23 Harald Reindl 2016-12-03 12:23:24 UTC
looks good and the also "python2-pyspf" can be removed afterwards which is important to be able uninstall finally python2, postfix config needs to explizit use "python3" since /usr/bin/python is stll python2 and on machines where python2 is uninstalled there is no /usr/bin/python at all but python3
___________________________________________________

[root@testserver:~]$ cat /etc/postfix/master.cf | grep spf
spf-policy-info unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python3 /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf-noreject.conf
spf-policy      unix  -       n       n       -        0      spawn user=nobody argv=/usr/bin/python3 /usr/libexec/postfix/policyd-spf /etc/python-policyd-spf/policyd-spf.conf
___________________________________________________

[root@testserver:~]$ rpm -qa | grep -i spf
perl-Mail-SPF-2.9.0-8.fc24.noarch
pypolicyd-spf-1.3.2-4.fc24.noarch
python3-pyspf-2.0.12-1.fc24.noarch
___________________________________________________

Dec  3 13:18:10 testserver postfix/smtpd[30101]: connect from local.rhsoft.net[62.178.103.85]
Dec  3 13:18:12 testserver policyd-spf[30109]: Pass; identity=helo; client-ip=62.178.103.85; helo=srv-rhsoft.rhsoft.net; envelope-from=h.reindl@thelounge.net; receiver=rhsoft@test.rh
Dec  3 13:18:12 testserver policyd-spf[30109]: Fail; identity=mailfrom; client-ip=62.178.103.85; helo=srv-rhsoft.rhsoft.net; envelope-from=h.reindl@thelounge.net; receiver=rhsoft@test.rh
Dec  3 13:18:12 testserver postfix/smtpd[30101]: NOQUEUE: reject: RCPT from local.rhsoft.net[62.178.103.85]: 550 5.7.1 <rhsoft@test.rh>: Recipient address rejected: Message rejected due to: SPF fail - not authorized. Please see http://www.openspf.net/Why?s=mfrom;id=h.reindl@thelounge.net;ip=62.178.103.85;r=rhsoft@test.rh; from=<h.reindl@thelounge.net> to=<rhsoft@test.rh> proto=ESMTP helo=<srv-rhsoft.rhsoft.net>
Dec  3 13:18:12 testserver postfix/smtpd[30101]: lost connection after RCPT from local.rhsoft.net[62.178.103.85]
Dec  3 13:18:12 testserver postfix/smtpd[30101]: disconnect from local.rhsoft.net[62.178.103.85] ehlo=1 mail=1 rcpt=0/1 commands=2/3

Comment 24 Harald Reindl 2016-12-07 11:12:51 UTC
i guess that only becomes more visible with Python3

"OSError: [Errno 98] Address already in use" - why does that stuff deal at it's own with the source-port? on a busy machine running dns-resolvers and all sort of milters and deamons for mail-processing that is racy by definition

Dec  6 12:03:20 mail-gw policyd-spf[8410]: Traceback (most recent call last):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 198, in getSource
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.s.bind(('', source_port))
Dec  6 12:03:20 mail-gw policyd-spf[8410]: OSError: [Errno 98] Address already in use
Dec  6 12:03:20 mail-gw policyd-spf[8410]: 
Dec  6 12:03:20 mail-gw policyd-spf[8410]: During handling of the above exception, another exception occurred:
Dec  6 12:03:20 mail-gw policyd-spf[8410]: Traceback (most recent call last):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/libexec/postfix/policyd-spf", line 707, in <module>
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    instance_dict, configData, peruser)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/libexec/postfix/policyd-spf", line 419, in _spfcheck
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    res = spf.check2(ip, helo_fake_sender, helo, querytime=configData.get('Lookup_Time'))
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 304, in check2
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 569, in check
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    rc = self.check1(spf, self.d, 0)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 608, in check1
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    return self.check0(spf, recursion)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 917, in check0
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    if self.cidrmatch(self.dns_a(arg,self.A), cidrlength):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 1222, in dns_a
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    r = self.dns(domainname, A)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 1310, in dns
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    for k, v in DNSLookup(name, qtype, self.strict, timeout):
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/spf.py", line 127, in DNSLookup
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    resp = req.req()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 244, in req
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.sendUDPRequest(server)
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 269, in sendUDPRequest
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.conn()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 205, in conn
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    self.getSource()
Dec  6 12:03:20 mail-gw policyd-spf[8410]:  File "/usr/lib/python3.5/site-packages/DNS/Base.py", line 202, in getSource
Dec  6 12:03:20 mail-gw policyd-spf[8410]:    if msg[0] != 98: raise
Dec  6 12:03:20 mail-gw policyd-spf[8410]: TypeError: 'OSError' object is not subscriptable

Comment 25 Bojan Smojver 2016-12-07 23:29:32 UTC
(In reply to Harald Reindl from comment #24)
> i guess that only becomes more visible with Python3
> 
> "OSError: [Errno 98] Address already in use" - why does that stuff deal at
> it's own with the source-port? on a busy machine running dns-resolvers and
> all sort of milters and deamons for mail-processing that is racy by
> definition

I think you should open an upstream bug for this (or at least another one in Red Hat Bugzilla). I haven't personally seen this, but then my config may be different.

Comment 26 Harald Reindl 2016-12-22 17:19:06 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1408273

Comment 27 Fedora Update System 2017-01-18 04:19:16 UTC
pypolicyd-spf-2.0.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e

Comment 28 Fedora Update System 2017-01-18 04:19:27 UTC
pypolicyd-spf-2.0.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7

Comment 29 Harald Reindl 2017-01-18 11:02:09 UTC
pypolicyd-spf-2.0.1-1.fc24 has a regression 

look at the receiver=<UNKNOWN> that is now on every logline and header besides that the logformat slightly changed

Jan 18 11:38:25 mail-gw policyd-spf[27152]: prepend Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=66.220.144.147; helo=mx-out.facebook.com; envelope-from=****; receiver=<UNKNOWN>

Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=62.210.161.103;
 helo=r103.p3.neolane.net; envelope-from****;
 receiver=<UNKNOWN>

Comment 30 Bojan Smojver 2017-01-18 11:44:11 UTC
(In reply to Harald Reindl from comment #29)
> pypolicyd-spf-2.0.1-1.fc24 has a regression 
> 
> look at the receiver=<UNKNOWN> that is now on every logline and header
> besides that the logformat slightly changed
> 
> Jan 18 11:38:25 mail-gw policyd-spf[27152]: prepend Received-SPF: Pass
> (mailfrom) identity=mailfrom; client-ip=66.220.144.147;
> helo=mx-out.facebook.com; envelope-from=****; receiver=<UNKNOWN>
> 
> Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=62.210.161.103;
>  helo=r103.p3.neolane.net; envelope-from****;
>  receiver=<UNKNOWN>

From the documentation of 2.0.1:

# In order to avoid disclosing BCC recipients in SPF header fields,
# Hide_Receiver is set to Yes by default in the interest of maximizing
# privacy.  This setting will replace the actual recipient with <UNKNOWN> both
# in header fields and SMTP responses.  The latter may make it more difficult
# for senders to troubleshoot issues with their SPF deployments.
#Hide_Receiver = No
Hide_Receiver = Yes

Comment 31 Harald Reindl 2017-01-18 11:55:16 UTC
thanks - i turn my karma to +1 with that option

Comment 32 Fedora Update System 2017-01-19 07:23:19 UTC
pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6e287bbdc7

Comment 33 Fedora Update System 2017-01-19 09:10:54 UTC
pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ec45fbc96e

Comment 34 Harald Reindl 2017-01-19 14:36:33 UTC
BTW: can someone ping the "logwatch" maintainers - the unnecessary change of logging to "prepend Received-SPF: " leads to blow all entries into **Unmatched Entries** in the daily logwatch mails - i would really love when developers stop fix things which ain't broken

Comment 35 Harald Reindl 2017-01-19 14:40:26 UTC
prepend Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=217.158.205.98; helo=***; envelope-from=***; receiver=*** also breaks scripts doing simple things like "/usr/bin/grep 'policyd-spf' /var/log/maillog | /usr/bin/grep 'Pass; identity=mailfrom'" as exmaple used in some helper scripts i had written long ago on our inbound server to filter whitelist_auth candidates for SpamAssassin

i could puke!

Comment 36 Bojan Smojver 2017-01-19 23:25:51 UTC
(In reply to Harald Reindl from comment #34)
> BTW: can someone ping the "logwatch" maintainers

Probably best if you open a logwatch bug in Fedora. Maintainers will then take it from there.

Comment 37 Fedora Update System 2017-01-29 00:20:10 UTC
pypolicyd-spf-2.0.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 38 Fedora Update System 2017-01-29 00:49:16 UTC
pypolicyd-spf-2.0.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 39 Harald Reindl 2017-02-04 16:33:14 UTC
see https://bugzilla.redhat.com/show_bug.cgi?id=1419274 because logwatch and could somebody ask upstream if they are crazy - there is a similar dicussion on isc-bind list because they introduced a new field for debugging purpose in the querylog breaking all sorts of analyze software


Note You need to log in before you can comment on or make changes to this bug.