Bug 1430020 - Update python-idstools to v0.5.6.
Summary: Update python-idstools to v0.5.6.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: python-idstools
Version: epel7
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: marcindulak
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-07 16:41 UTC by Jason Ish
Modified: 2017-04-03 01:19 UTC (History)
1 user (show)

Fixed In Version: python-idstools-0.5.6-5.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-11 15:18:15 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Patch to update epel7 package to v0.5.6. (1.14 KB, application/mbox)
2017-03-07 16:41 UTC, Jason Ish
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github https://github.com/jasonish py-idstools issues 40 0 None None None 2017-03-07 16:41:55 UTC

Description Jason Ish 2017-03-07 16:41:55 UTC
Created attachment 1260884 [details]
Patch to update epel7 package to v0.5.6.

Description of problem:

python-idstools 0.5.4 currently in EPEL has a bug causing some code paths to crash on Python 3.4. As the tools (ie: /usr/bin/idstools-rulecat) are only installed with the python34 variant of the package, some of these tools (like idstools-rulecat) are unusable.

0.5.6 fixes this issue.


How reproducible:
Reproducible for anyone trying to run idstools-rulecat and has Suricata installed.

Steps to Reproduce:
1. Install suricata (yum install suricata).
2. Install idstools (yum install python34-idstools)
3. Run idstools-rulecat (idstools-rulecat).

Actual results:
[root@9d3d1f76ead0 ~]# idstools-rulecat 
Traceback (most recent call last):
  File "/usr/bin/idstools-rulecat", line 12, in <module>
    sys.exit(main())
  File "/usr/lib/python3.4/site-packages/idstools/scripts/rulecat.py", line 725, in main
    args.url.append(resolve_etopen_url(args.suricata))
  File "/usr/lib/python3.4/site-packages/idstools/scripts/rulecat.py", line 642, in resolve_etopen_url
    suricata_version = idstools.suricata.get_version(suricata_path)
  File "/usr/lib/python3.4/site-packages/idstools/suricata.py", line 61, in get_version
    m = re.search("version ((\d+)\.(\d+)\.?(\d+|\w+)?)", output.strip())
  File "/usr/lib64/python3.4/re.py", line 170, in search
    return _compile(pattern, flags).search(string)
TypeError: can't use a string pattern on a bytes-like object


Expected results:
A successful run of idstools-rulecat.

Additional info:

Also, its a little frustrating that "python-idstools" resolves to "python2-idstools" which is the package that doesn't actually provide the tools, just the library.

Comment 1 marcindulak 2017-03-07 17:11:44 UTC
Hi,

that's the result of my reading of https://fedoraproject.org/wiki/Packaging:Python

"For other executables, the general rule is:

If only one executable is to be shipped, then it owns its own slot and should use /usr/bin/python3 from Fedora 22 on."

I decided to ship only python3 executables (scripts), but python2 can be run
with python2 `which idstools-rulecat`

If I read this policy correctly this may be a way of pushing python3 instead of python2.

See also this thread https://lists.fedoraproject.org/pipermail/packaging/2014-December/010360.html

The packaging rules have become complicated, especially if one wants to package for both Fedora and EPEL using the same spec file.

Comment 3 Jason Ish 2017-03-07 17:20:45 UTC
Fair enough. So from that I see that the executables need to go in the Python 3 package.

But the more generic "python-<packagename>" resolves to the system Python version which is currently version 2 across the board...  Sigh, thats a bit of UX issue I suppose.

Oh well, I'll be happy when its updated so the executable works with Python 3.

Thanks for the explanation.

Comment 4 marcindulak 2017-03-07 21:10:16 UTC
I don't reproduce your error after `yum install -y python34-idstools`:

$ strace idstools-u2json 2>&1 | grep 'bin/python'
readlink("/usr/bin/python3.4", 0x7ffde8806060, 4096) = -1 EINVAL (Invalid argument)
lstat("/usr/bin/python3.4", {st_mode=S_IFREG|0755, st_size=11312, ...}) = 0
read(3, "#!/usr/bin/python3.4 -s\n#\n# Laun"..., 214) = 214
read(3, "#!/usr/bin/python3.4 -s\n#\n# Laun"..., 4096) = 236
read(3, "#!/usr/bin/python3.4 -s\n#\n# Laun"..., 4096) = 236

$ python --version
Python 2.7.5

idstools scripts come with python3.4 shebang so they should be run with python3 by default:

$ grep python `which idstools-rulecat`
#!/usr/bin/python3.4 -s

Can you verify this?

About the release - should I wait until 0.5.7?

Comment 5 Jason Ish 2017-03-07 21:11:58 UTC
Did you install Suricata from epel? The error occurs when Suricata is on the path, as idstools-rulecat will automatically try and parse the version.

idstools-u2json is unaffected.

I have no immediate plans for a 0.5.7.

Comment 6 marcindulak 2017-03-07 21:25:50 UTC
I think we need 0.5.7. It seems that the problem is related to re in python3:

# python3
Python 3.4.5 (default, Nov  9 2016, 16:24:59) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import re
>>> output = subprocess.check_output(["suricata", "-V"])
>>> re.search("version ((\d+)\.(\d+)\.?(\d+|\w+)?)", output.strip())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.4/re.py", line 170, in search
    return _compile(pattern, flags).search(string)
TypeError: can't use a string pattern on a bytes-like object
>>> re.search(b"version ((\d+)\.(\d+)\.?(\d+|\w+)?)", output.strip())
<_sre.SRE_Match object; span=(17, 30), match=b'version 3.2.1'>
>>> 

# python2
Python 2.7.5 (default, Nov  6 2016, 00:28:07) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import re
>>> output = subprocess.check_output(["suricata", "-V"])
>>> re.search(b"version ((\d+)\.(\d+)\.?(\d+|\w+)?)", output.strip())
<_sre.SRE_Match object at 0x7f52b7089b58>

Comment 7 Jason Ish 2017-03-07 21:34:05 UTC
That doesn't look like 0.5.6, the regex in 0.5.6 is:

    m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())

https://github.com/jasonish/py-idstools/blob/0.5.6/idstools/suricata.py#L51

Comment 8 marcindulak 2017-03-07 21:50:59 UTC
OK, I see - the 0.5.6 actually fixes the re problem.

Comment 9 Fedora Update System 2017-03-08 21:22:37 UTC
python-idstools-0.5.6-5.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-68959e2eb0

Comment 10 Fedora Update System 2017-03-08 21:26:55 UTC
python-idstools-0.5.6-5.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-15810df738

Comment 11 Fedora Update System 2017-03-08 21:31:55 UTC
python-idstools-0.5.6-5.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-56837cefd5

Comment 12 Fedora Update System 2017-03-08 21:38:28 UTC
python-idstools-0.5.6-5.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0e9bb68871

Comment 13 Fedora Update System 2017-03-09 14:26:29 UTC
python-idstools-0.5.6-5.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-56837cefd5

Comment 14 Fedora Update System 2017-03-09 14:58:45 UTC
python-idstools-0.5.6-5.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-15810df738

Comment 15 Fedora Update System 2017-03-09 16:48:38 UTC
python-idstools-0.5.6-5.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-0e9bb68871

Comment 16 Fedora Update System 2017-03-10 23:22:55 UTC
python-idstools-0.5.6-5.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-68959e2eb0

Comment 17 Fedora Update System 2017-03-11 15:18:15 UTC
python-idstools-0.5.6-5.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2017-04-02 20:52:51 UTC
python-idstools-0.5.6-5.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2017-04-02 20:58:57 UTC
python-idstools-0.5.6-5.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2017-04-03 01:19:30 UTC
python-idstools-0.5.6-5.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.