Description of problem: The recent libxml2 errata: http://rhn.redhat.com/errata/RHSA-2008-0836.html ..."fixes" the problem by strictly limiting entities to 500,000. This was almost instantly noticed in Fedora as yum-metadata-parser (via. createrepo) then died parsing the XML metadata (because it tends to use a lot of > etc. RHN currently _only_ supplies XML metadata to yum, so at any point if we hit 500,000 entities yum will just die. This is a regression of the API, and needs a better fix. Also even if we are extremly careful, so we can guarantee RHN doesn't produce this error ... we have no control over what customers are putting into satellite etc. Also, personally, I think this is a very bad fix for any customers who are using the libxml2 API. ... they now have to "know" if their document might have "too many" entities in it, and act accordingly. Version-Release number of selected component (if applicable): libxml2-2.6.16-12.3 How reproducible: Always
I assumed we wouldn't have hit it already, given it went through QA, but that was apparently optimistic... "yum makecache" ...or anything that downloads the other.xml (changelog data) kills yum due to this problem. This also means you can't download all packages from RHN and run createrepo on them anymore, as createrepo will die. As a quick hack I did: # zcat /var/cache/yum/rhel-x86_64-server-5/other.xml.gz | \ perl -nle ' while (/\&/g) { ++$tot; } END { print $tot; }' 1086853 ...which implies if we need to do a _quick hack_ to raise the limit, we can probably get away with about 1.5 million instead of 500,000.
I'm looking at getting better algorithms but it's excruciatingly hard. Same level as designing an OOM killer that will only capture offending processes ! Purely raising that limit is not a much better solution. You may change one knob or another but the variety of exhaustion possible are really harder to process than what you seems to think. Daniel
Daniel confirmed that if RHN metadata generation moves from named entities to character entities libxml2 will be happy, which seems at least a viable short term. fix to give Daniel some breathing room. Eg. instead of < the metadata would have <
Created attachment 315291 [details] RHEL-5 Patch corresponding to the new upstream fix
Created attachment 315294 [details] RHEL-4 Patch corresponding to the new upstream fix
Created attachment 315298 [details] RHEL-3 Patch corresponding to the new upstream fix
Any possibility to get an attachement of a compressed XML file generated for yum which broke the old version for testing purposes ? xmllint --nooent --noent yum-test.xml.gz should work to test since xmllint accepts gzipped compressed input on files, if the file still fail to parse either the bug is not fixed or the generated XML is broken ;-) Daniel
Created attachment 315310 [details] RHN rpm ChangeLog data, from 2008-08-28. Has large number of named entities Sure, pretty much any other.xml.gz will probably do as a test. Here's the current one (bzip2'd to get around the BZ upload limits):
Thanks James, 117 MBytes that starts to be a nice beast, once gzipped again it makes a good regression test: wei:~/XML -> time xmllint --noent --stream ../yum.xml.gz real 0m4.856s user 0m4.833s sys 0m0.020s wei:~/XML -> cat .memdump 09:39:28 AM MEMORY ALLOCATED : 0, MAX was 63738 BLOCK NUMBER SIZE TYPE wei:~/XML -> No error indicates parsing went well... and with my debug version configured to track memory allocation i see no lost block and a very reasonable memory consumption. Maybe we need to add this in our regression tests somehow (maybe using valgrind instead to track potential memory leaks) For the make check tests upstream I added a program generating a million line document each line using entities with a bit of recursion, so any similar error will be caught before being pushed in the future, thanks again, and sorry for the troubles, Daniel
Created attachment 315351 [details] RHEL-2.1 Patch corresponding to the new upstream fix This still requires to increase the size of the entity structure. For libxml2-2.4.x there is really no placeholder to collect the required informations.
Created attachment 315476 [details] first test for bad behaviour
Created attachment 315477 [details] second test for bad behaviour
Created attachment 315478 [details] dtd for 3rd test pf bad behaviour
Created attachment 315479 [details] third test of bad behaviour
Created attachment 315480 [details] fourth test of bad behaviour
Created attachment 315481 [details] fifth test of bad behaviour
Created attachment 315482 [details] sixth test of bad behaviour
To use the six tests i just added save them on a local file system as well as the lol3.dtd, and try in sequence xmllint --noent --noout --loaddtd lolX.xml the program should return nearly immediately not consume much memory and raise at least one error about "Detected an entity reference loop" Daniel
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2008-0878.html