Bug 964940 - [RFE] rubygem-rdiscount bundles libmarkdown
[RFE] rubygem-rdiscount bundles libmarkdown
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: rubygem-rdiscount (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Gerd Pokorra
Fedora Extras Quality Assurance
: FutureFeature, Reopened
Depends On:
Blocks: DuplicSysLibsTracker
  Show dependency treegraph
 
Reported: 2013-05-19 18:07 EDT by Craig Barnes
Modified: 2015-02-07 03:16 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-02-07 03:16:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Craig Barnes 2013-05-19 18:07:42 EDT
The Source0 field of rubygem-rdiscount.spec[1] points to http://rubygems.org/gems/rdiscount-2.0.7.gem. The gem at that location has the Discount libmarkdown library bundled in the "ext" directory and it appears that the package compiles/links/installs it. Unless I'm mistaken, that's breaking the "no bundled libraries"[2] rule.

I've submitted a packaging request for Discount and hopefully it will be in Fedora soon. Perhaps the rubygem-rdiscount package should be patched to link against that, when it becomes available?


[1]: http://pkgs.fedoraproject.org/cgit/rubygem-rdiscount.git/tree/rubygem-rdiscount.spec#n10
[2]: https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries
Comment 1 Craig Barnes 2013-05-21 21:02:45 EDT
I've just noticed that rdiscount also bundles the markdown.7 man page from Discount too, so when trying to install both the rubygem-rdiscount package and the (now accepted and pending update) discount package, I get the following error:

Transaction Check Error:
  file /usr/share/man/man7/markdown.7.gz from install of rubygem-rdiscount-2.0.7-2.fc18.i686 conflicts with file from package discount-2.1.6-1.fc18.i686
Comment 2 Gerd Pokorra 2013-05-22 02:28:04 EDT
Sorry! I have had only limited access to the internet the last two days. So I will look for this problem today.
Comment 3 Gerd Pokorra 2013-05-22 03:59:48 EDT
I just builded rubygem-rdiscount with the new source 2.0.7.3 for rawhide and excluding the man-page /usr/share/man/man7/markdown.7.gz.

http://koji.fedoraproject.org/koji/taskinfo?taskID=5408354

Did it help or are they any other conflicts?
Comment 4 Gerd Pokorra 2013-05-22 06:40:55 EDT
Build of rubygem-rdiscount-2.0.7.3-1 are also down for F18 and F19 und in the update stream.
Comment 5 Craig Barnes 2013-05-22 14:30:07 EDT
That fixes the immediate problem of clashing files, yes. Thanks for that.

The problem of the bundled library still remains though. From what I can tell, /usr/lib/gems/ruby/rdiscount-2.0.7.3/lib/rdiscount.so contains an internal copy of the Discount library code (libmarkdown), instead of being dynamically linked against it.

I think the normal thing to do is either to patch the build system to link against the packaged libmarkdown or try to get an bundling exception.[1]

I would volunteer to do the patching, but the rdiscount developer seems to have thrown away Discount's configure script and Makefile, in favor of a Rakefile, which I can't read.

[1]: https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries#Exceptions
Comment 6 Gerd Pokorra 2013-05-23 02:03:11 EDT
I would like to try if I could build/link rubygem-rdiscount with/again discount and libmarkdown but I did not find a package in Fedora 18.

I found the review request of discount:
https://bugzilla.redhat.com/show_bug.cgi?id=740160

and a build of it for Fedora 18:
https://admin.fedoraproject.org/updates/discount-2.1.6-1.fc18

but nothing seems to be pushed into Fedora 18:

> yum search discount libmarkdown
Loaded plugins: langpacks, presto, refresh-packagekit
============================ N/S Matched: discount ============================
rubygem-rdiscount-doc.noarch : Documentation for rubygem-rdiscount
rubygem-rdiscount.x86_64 : Fast Implementation of Gruber's Markdown in C

  Name and summary matches mostly, use "search all" for everything.
Warning: No matches found for: libmarkdown
Comment 7 Gerd Pokorra 2013-05-23 06:45:21 EDT
I build your package discount and my package rubygem-rdiscount on a Fedora 17 system. Simple linking  rdiscount.so to /usr/lib64/libmarkdown.so.2.1.6 did not work.

./ruby.prog 
/usr/share/rubygems/rubygems/custom_require.rb:60:in `require': /usr/lib64/gems/exts/rdiscount-2.0.7.3/lib/rdiscount.so: undefined symbol: Init_rdiscount - /usr/lib64/gems/exts/rdiscount-2.0.7.3/lib/rdiscount.so (LoadError)
	from /usr/share/rubygems/rubygems/custom_require.rb:60:in `rescue in require'
	from /usr/share/rubygems/rubygems/custom_require.rb:35:in `require'
	from /usr/share/gems/gems/rdiscount-2.0.7.3/lib/rdiscount.rb:100:in `<top (required)>'
	from /usr/share/rubygems/rubygems/custom_require.rb:60:in `require'
	from /usr/share/rubygems/rubygems/custom_require.rb:60:in `rescue in require'
	from /usr/share/rubygems/rubygems/custom_require.rb:35:in `require'
	from ./ruby.prog:3:in `<main>'
Comment 8 Gerd Pokorra 2013-05-23 06:52:56 EDT
Output:

[gz016@vdesk1 ruby]$ nm -D /usr/lib64/gems/exts/rdiscount-2.0.7.3/lib/rdiscount.so.orig | grep -i init
00000000000065b0 T Init_rdiscount
000000000000b240 T ___mkd_initmmiot
0000000000002a58 T _init
0000000000005ee0 T mkd_initialize
[gz016@vdesk1 ruby]$ nm -D /usr/lib64/libmarkdown.so.2.1.6 | grep -i init
000000000000a660 T ___mkd_initmmiot
0000000000002b10 T _init
000000000000bfa0 T mkd_initialize
[gz016@vdesk1 ruby]$
Comment 9 Gerd Pokorra 2013-05-23 10:20:35 EDT
I do not think the gem really bundle the Discount libmarkdown library. It only use some of the same source files. But I think it will not be easy to change the ruby build process (gem install ...).

You can go in the ext directory and enter:

ruby extconf.rb

This will generate a Makefile to generate rdiscount.so.

I think it will not be easy to patch this Makefile to not use some C-source files and rather to link against libmarkdown.

How would this Makefile look like?

Indeed a better extconf.rb would needed to be written.
Comment 10 Gerd Pokorra 2013-05-24 04:38:18 EDT
I made a deeper look in the sources. rubygem-rdiscount take some C files from discount that have exactly the same data content. Other C files differ a like bit.

So I added this line in Rakefile that delete the files with the same content.

rm_f FileList['ext/amalloc.c', 'ext/amalloc.h', 'ext/basename.c', 'ext/css.c', 'ext/docheader.c', 'ext/dumptree.c', 'ext/emmatch.c', 'ext/resource.c', 'ext/xml.c', 'ext/xmlpage.c']

So this objects form this files will not be add to rdiscount.so.

I changed the Makefile to add the option: -lmarkdown

Line in Makefile
LOCAL_LIBS = -lmarkdown

build:
rm -f rdiscount.so
gcc -shared -o rdiscount.so Csio.o version.o markdown.o tags.o setup.o rdiscount.o html5.o generate.o toc.o flags.o mkdio.o -L. -L/usr/lib64 -L. -Wl,-z,relro -rdynamic -Wl,-export-dynamic  -m64 -lmarkdown -lruby  -lpthread -lrt -ldl -lcrypt -lm   -lc


running a test program with the new
 /usr/lib64/gems/exts/rdiscount-2.0.7.3/lib/rdiscount.so
library crashes:

 ./ruby.prog 
*** glibc detected *** /usr/bin/ruby: realloc(): invalid pointer: 0x0000000001347d48 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3732c7b616]
/lib64/libc.so.6(realloc+0x346)[0x3732c805b6]
/usr/lib64/gems/exts/rdiscount-2.0.7.3/lib/rdiscount.so(mkd_document+0xca)[0x7f9410ecd48a]
/usr/lib64/gems/exts/rdiscount-2.0.7.3/lib/rdiscount.so(+0x595d)[0x7f9410ec995d]
/lib64/libruby.so.1.9[0x39aab50533]
/lib64/libruby.so.1.9[0x39aab471a8]
...


So I think it makes no sense to patch the rdiscount sources.
Comment 11 Gerd Pokorra 2013-05-25 06:31:49 EDT
I see no way to link against the packaged libmarkdown. The rdiscount.so library is only a bytes greater with including the same code so I see no need to ask for an exception. I think this ticket can be closed.
Comment 12 Gerd Pokorra 2013-06-12 09:45:45 EDT
RDiscount is (effectively) statically linked against Discount, since it 
bundles it as part of the build process.

I do not, unfortunately. My limited knowledge around Ruby C extensions 
suggests that they have to (or are at least designed to be) only built 
from source.
Comment 13 Simon 2013-06-12 11:57:32 EDT
reopened
Please ask fesco for an exception.
Comment 14 Gerd Pokorra 2013-06-13 06:15:30 EDT
Okay! I only want to understand it.

Some more information:

At a 64bit system with the install sub package "libmarkdown(-devel)" of discount and the installed package "rubygem-rdiscount" you have this shared object libraries installed:

libmarkdown-rpm:

/usr/lib64/libmarkdown.so
/usr/lib64/libmarkdown.so.2.1.6

rubygem-rdiscount-rpm:

/usr/lib64/gems/exts/rdiscount-2.0.7.3/lib/rdiscount.so

Both libraries depend on building from some same C source files. Is this bundling libraries?

What is the problem?

The rdiscount library would only be loaded by a ruby program.

require 'rdiscount'
markdown = RDiscount.new( "Hello World!", :filter_html)
....
Comment 15 Craig Barnes 2013-06-13 06:34:32 EDT
The Markdown parsing in rdiscount is done by libmarkdown. libmarkdown is a library. So I'm afraid that's bundling libraries, yes.

The problems with bundling are well enumerated on the wiki page. I think the point of asking for an exception is to ensure that it's well justified and that there's a record of known cases.

I don't think upstream pays much attention to API/ABI compatibility except with the code they've dumped in their tree, so I think you're probably right about it being troublesome to patch and maintain.
Comment 16 Vít Ondruch 2013-06-14 06:43:25 EDT
Gerd, would you mind to discuss this issue with upstream? I.e. if they could link against system libmarkdown if available? I think this is the most natural think to do.
Comment 17 Gerd Pokorra 2013-06-14 09:17:38 EDT
I have already discussed this issue with one of gem owners (upstream).

Comment 12 includes parts of the answer from David Forst.

To find a solution I ask for an exception:

https://fedorahosted.org/fpc/ticket/304
Comment 18 Fedora End Of Life 2013-09-16 12:42:08 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 20 development cycle.
Changing version to '20'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora20
Comment 19 Ken Dreyer 2013-12-30 20:56:03 EST
Moving this bug back to Rawhide and marking as RFE to avoid auto-closing.
Comment 20 Mamoru TASAKA 2015-01-29 00:51:35 EST
(In reply to Gerd Pokorra from comment #10)
This segv is because ext/mkdio.h (in rubygem-rdiscount) says "typedef unsigned int mkd_flag_t;" where /usr/include/mkdio.h (included in libmarkdown-devel( says "typedef unsigned long mkd_flag_t;" and on 64bit size differs between int and long.

i.e. removing ext/mkdio.h from rubygem-rdiscount will remove this segv.

Note that with using system libmarkdown, two tests will fail. With make these
two tests succeed, another configure option needs to be added when compiling
discount (--with-dl=Both and --with-github-tags).

Now rubygem-rdiscount-2.1.7.1-7.fc2{2,1} uses system libmarkdown.so.
Comment 21 Fedora Update System 2015-01-29 00:55:13 EST
rubygem-rdiscount-2.1.7.1-7.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/rubygem-rdiscount-2.1.7.1-7.fc21
Comment 22 Fedora Update System 2015-02-06 23:04:18 EST
rubygem-rdiscount-2.1.7.1-7.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.