I updated the package on rawhide (i also added python3 variant), could you check that it solves your issue ?
Since some 5.1 tests fail, i feel uneasy pushing it in current stable releases, do you also need to get this fixed in F15/16 too ?

It would be pretty usual to keep this "in Rawhide only" for some time before publishing it as a Test Update for F16/F15. It's also possible to turn off karma automatism in bodhi, so no testers would manage to rush out an update without you having the final decision.

Also don't forget "repoquery --whatrequires python-feedparser", so obviously some testing will be necessary.

(In reply to comment #1)
> I updated the package on rawhide (i also added python3 variant), could you
> check that it solves your issue ?
> Since some 5.1 tests fail, i feel uneasy pushing it in current stable releases,
> do you also need to get this fixed in F15/16 too ?

This seems to fix it. With the earlier version, gPodder only used the feed URL and not the name of the podcast. With 5.1, gPodder uses the feed/podcast name.

I would like to have this version in F15 and F16, too. I agree with Michael, the update should be done carefully and with lots of testing.

Good, I need some time to do more testing (at least much more than running few scripts, and test suite) and decide what do : backport, backport fix,... I'm a bit worried about that test suite issue, I'll check this when I'll come back from fosdem
Sorry for being brief, I'm on my phone

Wut. What tests are failing? In Python 2 or 3? If you tell me, I'll work to fix it. If it's related to the compression tests not being included in the 5.1 release I can work with you to resolve that as well (it's fixed in SVN), or push out a new release if there's a requirement that the RPM be based on a release tarball.

> What tests are failing? In Python 2 or 3? 

Both, captain !

> If it's related to the compression tests not being included in the 5.1
release I can work with you to resolve that as well (it's fixed in SVN)

some of the failing tests are indeed related to compression (ie: test_zlib_bad) and unicode. others were failing as the test script was launched from parent directory (as it was done previously) so the relative path to tests files were broken (mostly http ones).

Here's a link to a scratch build logs with tests enabled:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3771136&name=build.log

I won't ask you for a release, if you think that current svn is not production ready or if you need more time just for broken tests.

Another scratch build using svn revision 680 (and executing tests from script current directory) which indeed fixes compressions tests issues but some still fails
http://koji.fedoraproject.org/koji/getfile?taskID=3771315&name=build.log

> with tests enabled

If the same tests succeed outside koji, it could be helpful to keep the %check section enabled but with || : overriding the tests' exit code. That way there would always be test results in the latest build.log.

Alternatively, you could make the %check section conditional, so it could be enabled conveniently for local builds:

%bcond_with tests

%if %{with tests}
...
%check
...
%endif


$ rpmbuild --rebuild --with tests python-feedparser-*.src.rpm

> it could be helpful to keep the %check section enabled but with || :

It still fails outside the chroot but it's a better solution than commenting out the %check section. As a maintainer, i'd prefer keeping the tests output in logs than offering a conditional. 

@Kurt: what's your take ? shipping r680 until you're done or wait upcoming release ?

Okay, the error in the r680 tests I'm aware of but I'm not able to reproduce. The maintainer for another distribution reported both of the errors (the UnicodeEncodeError and the date/time related test) on the feedparser issue tracker:

https://code.google.com/p/feedparser/issues/detail?id=314
https://code.google.com/p/feedparser/issues/detail?id=326

It appears that CPython on x64 platforms is able to handle a datetime range that produces an OverflowError on x86 platforms (such as my machine). I need to modify the datetime test cases so that they're testing the expected behavior on both platforms. I don't think that that failed test should be a show stopper, however.

The other is more serious, and seems to stem from Python believing that the filesystem encoding is ASCII instead of UTF-8. When a Unicode string is passed to the parse() function, feedparser attempts to try opening a file with that Unicode string as the filename. If the string contains a Unicode character and Python believes that the filesystem is ASCII-based, it attempts to encode the string in ASCII, which produces the UnicodeEncodeError. I don't know how to change my environment from UTF-8 to ASCII to test this bug, however, and I haven't found anything by searching the internet.

If somebody can point out how to make Python believe that my filesystem is ASCII encoded rather than UTF-8 encoded, I can test and fix that particular bug!

@Haïkel: r680 is stable, so it could used. However, after botching the 5.1 release by excluding the compression test files I've heard from a few maintainers, and I'd like to make it up to them (and you!) by making a 5.1.1 release to simplify everybody's life. I expect I'll have time to resolve both issues linked above by the end of February, if that's soon enough?

I forgot that I can affect the filesystem encoding by changing the locale. Issue 326 (the UnicodeEncodeError problem) is fixed in r681. I'll try to have a fix for the datetime test problem shortly, and then an actual release to accommodate all of you awesome distro maintainers!

@Kurt: thanks for you dedication, i'll package latest svn snapshot since it does fix issues (not only failing tests) and i'll look forward an upcoming release.

I'll follow Michael excellent advice of overriding tests exit code so we can execute tests (and keep logs) without failing builds until it's fixed.

5.1.2 has been released and contains a security fix (dangerous XML ENTITY declarations could get through feedparser's filter if they were encoded in a non-ASCII-compatible character encoding. I strongly recommend using 5.1.2 or backporting the fix (which you can cherry-pick from svn r703).

Are there any remaining problems running the test cases? I was under the impression that I fixed everything, but it's been a while since I've looked at this...

Created attachment 586027 [details]
feedparser-5.1.2 build test results

The testsuite still creates failures.

I've pushed 5.1.2 for Rawhide, however.

And a build job submitted for Fedora 17:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4094112

What version of BeautifulSoup is installed? Is chardet installed? What version? Is libxml2 installed? I swear I've seen these Unicode-related failures before, but I can't reproduce them on my machine.

I can't guess why that bad pipe error is there, however; that hasn't happened on my machine, ever. :(

And, for the Python 3 failure, that simply looks like feedparsertest.py isn't being converted using 2to3 before it's run. feedparser.py is converted automatically during installation, but both it and feedparsertest.py need manual conversion to run the tests. Do you have any idea if that could be done using the setup.py file? I'm specifically wondering if the invocation

    $ python3 setup.py test

could automatically run the files through 2to3 before running the tests. That might simplify things for everyone.

python-BeautifulSoup-3.2.0-4.fc17.noarch
python-chardet-2.0.1-4.fc17.noarch
libxml2-2.7.8-7.fc17.x86_64
libxml2-python-2.7.8-7.fc17.x86_64

The chardet package still uses the old/non-existant URL
  http://chardet.feedparser.org
No idea yet whether/where the project has moved.

BeautifulSoup 3.2.0 is not the latest. The package changelog entry from Feb 3rd confuses me:
  - Add Python 3 package based on BeautifulSoup 4.
The web page lists 3.2.1 (February 16, 2012) and 4.0.5 (April 27, 2012)

[...]

Package list for the F-17 build in the Fedora build system:
http://koji.fedoraproject.org/koji/getfile?taskID=4094118&name=root.log

No BeautifulSoup and chardet installed there (and they have not been made dependencies). It ran fewer tests

  | Ran 4237 tests in 6.436s
  | OK

than the local build (4374 tests) and only failed for the bad pipe error:
http://koji.fedoraproject.org/koji/getfile?taskID=4094118&name=build.log

# repoquery --whatrequires python3-feedparser
#

So, nothing explicitly depends on that one [yet].

[...]

I see that feedparser _optionally_ imports BeautifulSoup, chardet and specifies libxml2 as its preferred XML parser. And these three also affect the test-suite. However, the python-feedparser.spec file does not add these packages as BuildRequires (for the test-suite) and not as run-time requirements either. I've added a "TODO" comment to the spec file.

With my local F-17 installation:

$ rpm -q --whatrequires python-chardet
python-kitchen-1.1.1-1.fc17.noarch

$ rpm -q --whatrequires libxml2-python
sos-2.2-3.fc17.noarch
gnome-doc-utils-0.20.10-2.fc17.noarch
setroubleshoot-server-3.1.11-1.fc17.x86_64
createrepo-0.9.9-11.fc17.noarch

$ rpm -q python-BeautifulSoup
$

So, 2/3 would be installed (and hence available to feedparser at run-time) anyway. But if they are not available at build-time, they would not be tested. And the testsuite finds failures *if* they are available.

(In reply to comment #12)
> 5.1.2 has been released and contains a security fix (dangerous XML ENTITY
> declarations could get through feedparser's filter if they were encoded in a
> non-ASCII-compatible character encoding. I strongly recommend using 5.1.2 or
> backporting the fix (which you can cherry-pick from svn r703).
> 
> Are there any remaining problems running the test cases? I was under the
> impression that I fixed everything, but it's been a while since I've looked
> at this...

	
Haïkel, can you comment on this?  We have the fixed version going into Fedora 17, and Fedora 15 doesn't concern me as much, but we really do need to see this fixed in Fedora 16.  Is there a problem, at this point, with upgrading to 5.1.2 on F16?  If there is, can you backport the fix?  See bug #824602 for further details (its parent is the CVE bug).  Thanks.

python-feedparser-5.1.2-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/python-feedparser-5.1.2-1.fc16

Package python-feedparser-5.1.2-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing python-feedparser-5.1.2-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-10550/python-feedparser-5.1.2-1.fc16
then log in and leave karma (feedback).

python-feedparser-5.1.2-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.