Bug 787401 - 5.1 has been released
Summary: 5.1 has been released
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-feedparser
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Haïkel Guémar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 753948
TreeView+ depends on / blocked
 
Reported: 2012-02-04 20:11 UTC by Ville-Pekka Vainio
Modified: 2012-08-05 21:30 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-05 21:30:47 UTC


Attachments (Terms of Use)
feedparser-5.1.2 build test results (17.69 KB, text/plain)
2012-05-22 12:58 UTC, Michael Schwendt
no flags Details

Description Ville-Pekka Vainio 2012-02-04 20:11:34 UTC
Feedparser 5.1 has been released: http://code.google.com/p/feedparser/
Could you please update the package. Apparently the new version would fix the following gPodder bug.

This Bugzilla: Bug 753948
gPodder upstream: https://bugs.gpodder.org/show_bug.cgi?id=1536
Feedparser upstream: http://code.google.com/p/feedparser/issues/detail?id=291 and http://code.google.com/p/feedparser/issues/detail?id=298

Comment 1 Haïkel Guémar 2012-02-04 23:35:58 UTC
I updated the package on rawhide (i also added python3 variant), could you check that it solves your issue ?
Since some 5.1 tests fail, i feel uneasy pushing it in current stable releases, do you also need to get this fixed in F15/16 too ?

Comment 2 Michael Schwendt 2012-02-05 09:14:10 UTC
It would be pretty usual to keep this "in Rawhide only" for some time before publishing it as a Test Update for F16/F15. It's also possible to turn off karma automatism in bodhi, so no testers would manage to rush out an update without you having the final decision.

Also don't forget "repoquery --whatrequires python-feedparser", so obviously some testing will be necessary.

Comment 3 Ville-Pekka Vainio 2012-02-05 09:50:22 UTC
(In reply to comment #1)
> I updated the package on rawhide (i also added python3 variant), could you
> check that it solves your issue ?
> Since some 5.1 tests fail, i feel uneasy pushing it in current stable releases,
> do you also need to get this fixed in F15/16 too ?

This seems to fix it. With the earlier version, gPodder only used the feed URL and not the name of the podcast. With 5.1, gPodder uses the feed/podcast name.

I would like to have this version in F15 and F16, too. I agree with Michael, the update should be done carefully and with lots of testing.

Comment 4 Haïkel Guémar 2012-02-05 11:30:11 UTC
Good, I need some time to do more testing (at least much more than running few scripts, and test suite) and decide what do : backport, backport fix,... I'm a bit worried about that test suite issue, I'll check this when I'll come back from fosdem
Sorry for being brief, I'm on my phone

Comment 5 Kurt McKee 2012-02-08 07:38:00 UTC
Wut. What tests are failing? In Python 2 or 3? If you tell me, I'll work to fix it. If it's related to the compression tests not being included in the 5.1 release I can work with you to resolve that as well (it's fixed in SVN), or push out a new release if there's a requirement that the RPM be based on a release tarball.

Comment 6 Haïkel Guémar 2012-02-08 10:03:06 UTC
> What tests are failing? In Python 2 or 3? 

Both, captain !

> If it's related to the compression tests not being included in the 5.1
release I can work with you to resolve that as well (it's fixed in SVN)

some of the failing tests are indeed related to compression (ie: test_zlib_bad) and unicode. others were failing as the test script was launched from parent directory (as it was done previously) so the relative path to tests files were broken (mostly http ones).

Here's a link to a scratch build logs with tests enabled:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3771136&name=build.log

I won't ask you for a release, if you think that current svn is not production ready or if you need more time just for broken tests.

Another scratch build using svn revision 680 (and executing tests from script current directory) which indeed fixes compressions tests issues but some still fails
http://koji.fedoraproject.org/koji/getfile?taskID=3771315&name=build.log

Comment 7 Michael Schwendt 2012-02-08 11:31:25 UTC
> with tests enabled

If the same tests succeed outside koji, it could be helpful to keep the %check section enabled but with || : overriding the tests' exit code. That way there would always be test results in the latest build.log.

Alternatively, you could make the %check section conditional, so it could be enabled conveniently for local builds:

%bcond_with tests

%if %{with tests}
...
%check
...
%endif


$ rpmbuild --rebuild --with tests python-feedparser-*.src.rpm

Comment 8 Haïkel Guémar 2012-02-08 14:37:55 UTC
> it could be helpful to keep the %check section enabled but with || :

It still fails outside the chroot but it's a better solution than commenting out the %check section. As a maintainer, i'd prefer keeping the tests output in logs than offering a conditional. 

@Kurt: what's your take ? shipping r680 until you're done or wait upcoming release ?

Comment 9 Kurt McKee 2012-02-09 08:36:31 UTC
Okay, the error in the r680 tests I'm aware of but I'm not able to reproduce. The maintainer for another distribution reported both of the errors (the UnicodeEncodeError and the date/time related test) on the feedparser issue tracker:

https://code.google.com/p/feedparser/issues/detail?id=314
https://code.google.com/p/feedparser/issues/detail?id=326

It appears that CPython on x64 platforms is able to handle a datetime range that produces an OverflowError on x86 platforms (such as my machine). I need to modify the datetime test cases so that they're testing the expected behavior on both platforms. I don't think that that failed test should be a show stopper, however.

The other is more serious, and seems to stem from Python believing that the filesystem encoding is ASCII instead of UTF-8. When a Unicode string is passed to the parse() function, feedparser attempts to try opening a file with that Unicode string as the filename. If the string contains a Unicode character and Python believes that the filesystem is ASCII-based, it attempts to encode the string in ASCII, which produces the UnicodeEncodeError. I don't know how to change my environment from UTF-8 to ASCII to test this bug, however, and I haven't found anything by searching the internet.

If somebody can point out how to make Python believe that my filesystem is ASCII encoded rather than UTF-8 encoded, I can test and fix that particular bug!

@Haïkel: r680 is stable, so it could used. However, after botching the 5.1 release by excluding the compression test files I've heard from a few maintainers, and I'd like to make it up to them (and you!) by making a 5.1.1 release to simplify everybody's life. I expect I'll have time to resolve both issues linked above by the end of February, if that's soon enough?

Comment 10 Kurt McKee 2012-02-09 09:33:42 UTC
I forgot that I can affect the filesystem encoding by changing the locale. Issue 326 (the UnicodeEncodeError problem) is fixed in r681. I'll try to have a fix for the datetime test problem shortly, and then an actual release to accommodate all of you awesome distro maintainers!

Comment 11 Haïkel Guémar 2012-02-09 18:23:47 UTC
@Kurt: thanks for you dedication, i'll package latest svn snapshot since it does fix issues (not only failing tests) and i'll look forward an upcoming release.

I'll follow Michael excellent advice of overriding tests exit code so we can execute tests (and keep logs) without failing builds until it's fixed.

Comment 12 Kurt McKee 2012-05-08 15:37:53 UTC
5.1.2 has been released and contains a security fix (dangerous XML ENTITY declarations could get through feedparser's filter if they were encoded in a non-ASCII-compatible character encoding. I strongly recommend using 5.1.2 or backporting the fix (which you can cherry-pick from svn r703).

Are there any remaining problems running the test cases? I was under the impression that I fixed everything, but it's been a while since I've looked at this...

Comment 13 Michael Schwendt 2012-05-22 12:58:17 UTC
Created attachment 586027 [details]
feedparser-5.1.2 build test results

The testsuite still creates failures.

I've pushed 5.1.2 for Rawhide, however.

Comment 14 Michael Schwendt 2012-05-22 13:09:34 UTC
And a build job submitted for Fedora 17:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4094112

Comment 15 Kurt McKee 2012-05-23 05:37:20 UTC
What version of BeautifulSoup is installed? Is chardet installed? What version? Is libxml2 installed? I swear I've seen these Unicode-related failures before, but I can't reproduce them on my machine.

I can't guess why that bad pipe error is there, however; that hasn't happened on my machine, ever. :(

And, for the Python 3 failure, that simply looks like feedparsertest.py isn't being converted using 2to3 before it's run. feedparser.py is converted automatically during installation, but both it and feedparsertest.py need manual conversion to run the tests. Do you have any idea if that could be done using the setup.py file? I'm specifically wondering if the invocation

    $ python3 setup.py test

could automatically run the files through 2to3 before running the tests. That might simplify things for everyone.

Comment 16 Michael Schwendt 2012-05-23 08:03:39 UTC
python-BeautifulSoup-3.2.0-4.fc17.noarch
python-chardet-2.0.1-4.fc17.noarch
libxml2-2.7.8-7.fc17.x86_64
libxml2-python-2.7.8-7.fc17.x86_64

The chardet package still uses the old/non-existant URL
  http://chardet.feedparser.org
No idea yet whether/where the project has moved.

BeautifulSoup 3.2.0 is not the latest. The package changelog entry from Feb 3rd confuses me:
  - Add Python 3 package based on BeautifulSoup 4.
The web page lists 3.2.1 (February 16, 2012) and 4.0.5 (April 27, 2012)

[...]

Package list for the F-17 build in the Fedora build system:
http://koji.fedoraproject.org/koji/getfile?taskID=4094118&name=root.log

No BeautifulSoup and chardet installed there (and they have not been made dependencies). It ran fewer tests

  | Ran 4237 tests in 6.436s
  | OK

than the local build (4374 tests) and only failed for the bad pipe error:
http://koji.fedoraproject.org/koji/getfile?taskID=4094118&name=build.log

Comment 17 Michael Schwendt 2012-05-23 10:35:06 UTC
# repoquery --whatrequires python3-feedparser
#

So, nothing explicitly depends on that one [yet].

[...]

I see that feedparser _optionally_ imports BeautifulSoup, chardet and specifies libxml2 as its preferred XML parser. And these three also affect the test-suite. However, the python-feedparser.spec file does not add these packages as BuildRequires (for the test-suite) and not as run-time requirements either. I've added a "TODO" comment to the spec file.

With my local F-17 installation:

$ rpm -q --whatrequires python-chardet
python-kitchen-1.1.1-1.fc17.noarch

$ rpm -q --whatrequires libxml2-python
sos-2.2-3.fc17.noarch
gnome-doc-utils-0.20.10-2.fc17.noarch
setroubleshoot-server-3.1.11-1.fc17.x86_64
createrepo-0.9.9-11.fc17.noarch

$ rpm -q python-BeautifulSoup
$

So, 2/3 would be installed (and hence available to feedparser at run-time) anyway. But if they are not available at build-time, they would not be tested. And the testsuite finds failures *if* they are available.

Comment 18 Vincent Danen 2012-05-23 22:44:02 UTC
(In reply to comment #12)
> 5.1.2 has been released and contains a security fix (dangerous XML ENTITY
> declarations could get through feedparser's filter if they were encoded in a
> non-ASCII-compatible character encoding. I strongly recommend using 5.1.2 or
> backporting the fix (which you can cherry-pick from svn r703).
> 
> Are there any remaining problems running the test cases? I was under the
> impression that I fixed everything, but it's been a while since I've looked
> at this...

	
Haïkel, can you comment on this?  We have the fixed version going into Fedora 17, and Fedora 15 doesn't concern me as much, but we really do need to see this fixed in Fedora 16.  Is there a problem, at this point, with upgrading to 5.1.2 on F16?  If there is, can you backport the fix?  See bug #824602 for further details (its parent is the CVE bug).  Thanks.

Comment 19 Fedora Update System 2012-07-10 20:30:58 UTC
python-feedparser-5.1.2-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/python-feedparser-5.1.2-1.fc16

Comment 20 Fedora Update System 2012-07-11 23:59:12 UTC
Package python-feedparser-5.1.2-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing python-feedparser-5.1.2-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-10550/python-feedparser-5.1.2-1.fc16
then log in and leave karma (feedback).

Comment 21 Fedora Update System 2012-08-05 21:30:47 UTC
python-feedparser-5.1.2-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.