Would you help packaging those parsers?

Those would be markdown, markdown_inline, lua, vimdoc and query

I'd be happy to help with packaging, though I'm not familiar with the official packaging procedures/requirements (as in, I've only built for copr thus far). Since I've been meaning to look into packaging some packages anyway (e.g. stylua and lua-language-sever), this might be a good starting point :)

Is there any particular guide on packaging tree sitter parsers that is a good starting point?

I'm also interested to give it a try. But don't take me too seriously until I share some specfile example or copr repo with an actual attempt ...

> Is there any particular guide on packaging tree sitter parsers that is a good starting point?

Looking at the packages available in Fedora, it seems that there is only one tree-sitter parser packaged:

https://packages.fedoraproject.org/pkgs/tree-sitter-java/libtree-sitter-java/

So I don't think there is any specific fedora policy beyond the naming scheme used here.

From the packaging perspective, it is sheer shared library:

```
$ rpm -ql libtree-sitter-java
/usr/lib/.build-id
/usr/lib/.build-id/31
/usr/lib/.build-id/31/bde20d7e4c1cc2f89a06469dcc4aa5345facec
/usr/lib64/libtree-sitter-java.so.0
/usr/lib64/libtree-sitter-java.so.0.0
/usr/share/doc/libtree-sitter-java
/usr/share/doc/libtree-sitter-java/README.md
/usr/share/licenses/libtree-sitter-java
/usr/share/licenses/libtree-sitter-java/LICENSE
```

So there is nothing much tree-sitter specific there.

When you check source code of those parsers, you notice that each is implemented in different combination of language and build system, so to package a parser, one will need to
understand packaging and fedora packaging policies in general, and how to apply that for given
language/build system.

Well, we would need to package them specifically for neovim. Other editors might ship parsers with the same name which don't work with neovim or vice versa.

List of parsers:
https://github.com/nvim-treesitter/nvim-treesitter?tab=readme-ov-file#supported-languages

We would need package names to be:

neovim-tree-sitter-lua
neovim-tree-sitter-markdown
neovim-tree-sitter-query
...

and install it in /usr/lib64/nvim/parsers

and add /usr/lib64/nvim to the runtime path to pick them up. We need a proof of concept package first. Probably best to do it in copr.


P.S: I created lua-languae-server package some time ago. Would be great if someone could pick that up and maintain it:
https://copr.fedorainfracloud.org/coprs/asn/lsp/packages/

I've been maintaining a reasonably up-to-date lua-language-server copr at https://copr.fedorainfracloud.org/coprs/yorickpeterse/lua-language-server/ for a while now as well. I'd be happy to somehow upstream that (and maybe StyLua) into Fedora at some point, but that's probably best to discuss separately :)

Regarding the tree sitter packages: I've added an entry to my calendar for next weekend (so the 8/9th of June) to take a look. I'll start with neovim-tree-sitter-lua, unless somebody else wants to take it up instead.

I took a brief look, but I have two questions:

1. Do we really need to name them "neovim-tree-sitter-lua" and such, instead of
   just libtree-sitter-lua (for example)? It seems e.g. Arch Linux just installs
   the packages as regular ones (i.e. "tree-sitter-lua" instead of making them
   Neovim specific). They then symbolically link the parsers into a directory
   that Neovim picks up: <https://gitlab.archlinux.org/archlinux/packaging/packages/neovim/-/blob/main/PKGBUILD?ref_type=heads#L82>

2. Neovim doesn't have /usr/lib64 in its `runtimepath` by default, but it does
   have /usr/lib (/usr/lib/nvim to be exact) in its path. The list of paths in
   `runtimepath` when using `nvim --clean` is as follows:

- /etc/xdg/nvim
- /var/home/USERNAME/.local/share/flatpak/exports/share/nvim/site
- /var/lib/flatpak/exports/share/nvim/site
- /usr/local/share/nvim/site
- /usr/share/nvim/site
- /usr/share/nvim/runtime
- /usr/share/nvim/runtime/pack/dist/opt/matchit
- /usr/lib/nvim
- /usr/share/nvim/site/after
- /usr/local/share/nvim/site/after
- /var/lib/flatpak/exports/share/nvim/site/after
- /var/home/USERNAME/.local/share/flatpak/exports/share/nvim/site/after
- /etc/xdg/nvim/after

> 1. Do we really need to name them "neovim-tree-sitter-lua" and such, instead of
>  just libtree-sitter-lua (for example)? 

I would also package them
in a generic way (as shown on the libtree-sitter-java example), and then add neovim
specific subpackage with a symlink or a config file (using vim.treesitter.language.add)
to make this grammar enabled by default in neovim. There could be other editors which
can use these grammars.

That said this assumes that the tree sitter grammars are not neovim specific, which
Andreas noted above that it's not always the case. So maybe we need to make this
decision for each grammar?

I looked at tree-sitter-markdown[1], at it mentions neovim just in it's
README among another editor:

> The main goal for this parser is to provide syntactical information for
> syntax highlighting in parsers such as neovim and helix.

[1] https://github.com/tree-sitter-grammars/tree-sitter-markdown

The grammars themselves are not specific to Neovim at all. All Neovim does is require specific versions per https://github.com/neovim/neovim/blob/b6c1ce8a9545575f8b1ea75e29a4ee9f018e3276/cmake.deps/deps.txt.

Neovim searches for parsers in `parser/LANG.so` relative to the directories in `runtimepath` (e.g. `/usr/lib/nvim/parser/lua.so`). Since /usr/lib isn't symlinked to /usr/lib64, I think we'd have to do the following:

1. Add /usr/lib64/nvim to "runtimepath" (perhaps using the sysinit.vim file shipped with the neovim package?)
2. Build the various tree-sitter grammars in a standalone fashion (i.e. libtree-sitter-lua, libtree-sitter-query, etc)
3. In the Neovim package, include a symbolic link to link /usr/lib64/libtree-sitter-X.so.0 into /usr/lib64/nvim/parser/X.so

An alternative approach is to skip the first step and link into /usr/share/nvim/runtime/parser/X.so similar to what the Arch Linux package does. I think this could be more reliable in case the sysinit.vim file somehow isn't loaded (not sure if you can actually skip that?).

Why do I want neovim-tree-sitter-<lang>?

Lets take a look a the markdown parser. The markdown parser change completely some time ago. You can see this here:

https://github.com/cryptomilk/nightcity.nvim/commit/5eb85eb2a7820eaa2f224dee5754e6ad41575509

Ok, so editor1 requires the old version of markdown but editor 2 requires the newer version. How do you fix that?

Then there is:

https://github.com/tree-sitter-grammars/tree-sitter-markdown and also
https://github.com/ikatyang/tree-sitter-markdown

What if editor1 requires the first and editor2 requires the second? How do we solve that?

tree-sitter parsers are normally specific to applications. That's why helix ships them as wasm binaries.

That makes sense, but in that case wouldn't it be much easier to just compile Neovim such that it builds the appropriate parsers for us (as in, what it does by default)? Perhaps I don't understand Fedora's packaging requirements correctly, but it seems a bit odd to extract a dependency as a separate package if its version is always tied to a specific Neovim version (i.e. if you upgrade one of them you must also upgrade the other).

@asn why not bundle them into the source rpm and install the markdown, markdown_inline, lua, vimdoc and query treesitter .so files into /usr/lib/nvim  (or some other runtime path) ?

Created attachment 2037958 [details]
neovim.spec patch to bundle the required tree-sitter parsers

I have tested this with fedpkg mockbuild and the generated rpms seem fine. Let me know what you think.

I don't like bundling. Those parsers might change more often then neovim so it makes sense to update them if needed or not. However we might go with libtree-sitter-<parser><major>. So we end up e.g. libtree-sitter-markdown0 and install in /usr/lib64/tree-sitter-markdown-0.

This way we have versioned directories and could in case install multiple versions in parallel. For neovim we could do symlinks or use update-alternatives.

What do you think?

If a user wishes to update the parsers, they can simply install them (e.g. using nvim-treesitter) in a directory that takes precedence (e.g. something in ~/.config/nvim or ~/.local) over the system-wide directory.

This means extracting these libraries into separate dependencies just adds unnecessary work, without it bringing any benefits. So unless Fedora's packaging requirements state a package can't bundle its dependencies, I feel just keeping them bundled is best for everybody.

openSUSE created tree-sitter-<lang> packages. They also created RPM macros for tree-sitter. See https://build.opensuse.org/package/show/editors/tree-sitter

We should do the same.

The macros need to be added to https://src.fedoraproject.org/rpms/rust-tree-sitter

Do you really think it is worth the work? It is unlikely that any other package will need these treesitter dependencies.

Bundling should normally be avoided, but it looks like I'm the only one interested in it.

I've cleaned up your patch and this is what I have right now:
https://src.fedoraproject.org/fork/asn/rpms/neovim/c/201d648f24e3eb078dd6705de592aef63d03ce03?branch=wip-treesitter-parser

This looks good to me :)

Thanks!

FEDORA-2024-508278d5ef (neovim-0.10.0-7.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-508278d5ef

FEDORA-2024-508278d5ef (neovim-0.10.0-7.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.

Thanks!

Can this be back ported on the f40 branch ?

FEDORA-2024-c8a6881eed (neovim-0.10.0-6.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-c8a6881eed

FEDORA-2024-c8a6881eed has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-c8a6881eed`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-c8a6881eed

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2024-c8a6881eed (neovim-0.10.0-6.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.

The issue is still present on Fedora 40 after updating to neovim-0.10.0-6.fc40.

```
Error detected while processing BufReadPost Autocommands for "*":
Error executing lua callback: /usr/share/nvim/runtime/filetype.lua:35: Error executing lua: /usr/share/nvim/runtime/filetype.lu
a:36: BufReadPost Autocommands for "*"..FileType Autocommands for "*"..function <SNR>1_LoadFTPlugin[20]..script /usr/share/nvim
/runtime/ftplugin/lua.lua: Vim(runtime):E5113: Error while calling lua chunk: /usr/share/nvim/runtime/lua/vim/treesitter/langua
ge.lua:107: no parser for 'lua' language, see :help treesitter-parsers
stack traceback:
        [C]: in function 'error'
        /usr/share/nvim/runtime/lua/vim/treesitter/language.lua:107: in function 'add'
        /usr/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:111: in function 'new'
        /usr/share/nvim/runtime/lua/vim/treesitter.lua:41: in function '_create_parser'
        /usr/share/nvim/runtime/lua/vim/treesitter.lua:108: in function 'get_parser'
        /usr/share/nvim/runtime/lua/vim/treesitter.lua:416: in function 'start'
        /usr/share/nvim/runtime/ftplugin/lua.lua:2: in main chunk
        [C]: in function 'nvim_cmd'
        /usr/share/nvim/runtime/filetype.lua:36: in function </usr/share/nvim/runtime/filetype.lua:35>
        [C]: in function 'nvim_buf_call'
        /usr/share/nvim/runtime/filetype.lua:35: in function </usr/share/nvim/runtime/filetype.lua:10>
stack traceback:
        [C]: in function 'nvim_cmd'
        /usr/share/nvim/runtime/filetype.lua:36: in function </usr/share/nvim/runtime/filetype.lua:35>
        [C]: in function 'nvim_buf_call'
        /usr/share/nvim/runtime/filetype.lua:35: in function </usr/share/nvim/runtime/filetype.lua:10>
stack traceback:
        [C]: in function 'nvim_buf_call'
        /usr/share/nvim/runtime/filetype.lua:35: in function </usr/share/nvim/runtime/filetype.lua:10>
```

Using a Fedora 40 container, I'm not able to reproduce the above error. I did the following:

1. podman run --interactive --tty --rm fedora:40 /bin/bash
2. dnf install neovim
3. nvim --clean test.lua

The parsers also appear to be available as expected:

    $ ls /usr/lib64/nvim/parser/
    bash.so  c.so  lua.so  markdown.so  markdown_inline.so  python.so  query.so  vim.so  vimdoc.so

The exact package version of Neovim is neovim-0.10.0-6.fc40 in this case.

I dug a little bit, and I can reproduce the issue consistently:

1. podman run --interactive --tty --rm fedora:40 /bin/bash
2. dnf install -y neovim git
3. mkdir -p ~/.config/nvim
4. Create `~/.config/nvim/init.lua` with the following content:

-- Bootstrap lazy.nvim
local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not (vim.uv or vim.loop).fs_stat(lazypath) then
  local lazyrepo = "https://github.com/folke/lazy.nvim.git"
  local out = vim.fn.system({ "git", "clone", "--filter=blob:none", "--branch=stable", lazyrepo, lazypath })
  if vim.v.shell_error ~= 0 then
    vim.api.nvim_echo({
      { "Failed to clone lazy.nvim:\n", "ErrorMsg" },
      { out, "WarningMsg" },
      { "\nPress any key to exit..." },
    }, true, {})
    vim.fn.getchar()
    os.exit(1)
  end
end
vim.opt.rtp:prepend(lazypath)
-- Make sure to setup `mapleader` and `maplocalleader` before
-- loading lazy.nvim so that mappings are correct.
-- This is also a good place to setup other settings (vim.opt)
vim.g.mapleader = " "
vim.g.maplocalleader = "\\"
-- Setup lazy.nvim
require("lazy").setup({
  spec = {
    -- add your plugins here
  },
  -- Configure any other settings here. See the documentation for more details.
  -- colorscheme that will be used when installing plugins.
  install = { colorscheme = { "habamax" } },
  -- automatically check for plugin updates
  checker = { enabled = true },
})

5. Open the file with Neovim again
6. Errors appear


This configuration file is the basic installation of Lazy.nvim as explained here: https://lazy.folke.io/installation
Note that this exact same configuration does NOT raise ANY error for Neovim 0.10 on ArchLinux or openSUSE Tumbleweed.

It appears that lazy.nvim is messing with the runtime path in a way that breaks
the Fedora package. Using `nvim --clean`, the output of `echo &runtimepath` is
the following:

    /etc/xdg/nvim
    /usr/local/share/nvim/site
    /usr/share/nvim/site
    /usr/share/nvim/runtime
    /usr/share/nvim/runtime/pack/dist/opt/matchit
    /usr/lib64/nvim
    /usr/share/nvim/site/after
    /usr/local/share/nvim/site/after
    /etc/xdg/nvim/after

Using just `nvim` without the init.lua, the output is this:

    /root/.config/nvim
    /etc/xdg/nvim
    /root/.local/share/nvim/site
    /usr/local/share/nvim/site
    /usr/share/nvim/site
    /usr/share/nvim/runtime
    /usr/share/nvim/runtime/pack/dist/opt/matchit
    /usr/lib64/nvim
    /usr/share/nvim/site/after
    /usr/local/share/nvim/site/after
    /root/.local/share/nvim/site/after
    /etc/xdg/nvim/after
    /root/.config/nvim/after
    /usr/share/vim/vimfiles/

Using the mentioned init.lua, the output is as follows instead:

    /root/.config/nvim
    /root/.local/share/nvim/site
    /root/.local/share/nvim/lazy/lazy.nvim
    /usr/share/nvim/runtime
    /usr/share/nvim/runtime/pack/dist/opt/matchit
    /usr/lib/nvim
    /root/.config/nvim/after
    /root/.local/state/nvim/lazy/readme

Here we can see that for reason lazy.nvim is removing /usr/lib64/nvim from the
runtimepath, resulting in the parsers not being found.

Per https://github.com/folke/lazy.nvim/issues/1343#issuecomment-2226493978, it
seems this is an issue specifically with lazy.nvim messing up `runtimepath`, and
as such should be fixed there.