Bug 1758837 - Sourcekit-lsp
Summary: Sourcekit-lsp
Alias: None
Product: Fedora
Classification: Fedora
Component: swift-lang
Version: rawhide
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Ron Olson
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2019-10-06 04:43 UTC by Ayman
Modified: 2020-01-04 20:07 UTC (History)
2 users (show)

Fixed In Version: swift-lang-5.1.3-0.1.20191213git005fc1f.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-01-04 20:07:48 UTC
Type: Bug

Attachments (Terms of Use)
screenshot of working sourcekit-lsp on fedora (135.48 KB, image/png)
2019-12-08 18:38 UTC, Ron Olson
no flags Details

Description Ayman 2019-10-06 04:43:18 UTC
Any plans on including `sourcekit-lsp`? The latest toolchains for Ubuntu includes it.

Comment 1 Ayman 2019-10-06 04:53:24 UTC
I couldn't get it to work correctly on Fedora 30. Like it compiles fine, but I get nothing out of the lsp, the returned results are always empty. In VSCode, i'm getting `failed to create SwiftPMWorkspace: cannotDetermineHostToolchain`.

Comment 2 Ron Olson 2019-10-06 09:53:12 UTC
Hmm, sure, let me give it a try. I'd be interested to know how to get it to work properly in case more tweaking has to be done in the packaging.

Comment 3 Ron Olson 2019-10-09 14:50:40 UTC
So I've been looking into this quite a bit; you're right in that it's included if Swift is built from the master branch on Github; It built properly on Fedora using F29, which still has Python 2. It is not part of the releases, however; if you look at https://github.com/apple/sourcekit-lsp/tags, there is no release or tag that is in sync with Swift's "-RELEASE". That plus its stated on the README that it's under heavy development, *and* the fact that I too have been unable to get it working at all, makes me worried that it would lead to a crummy user experience with Swift in general.

I'm going to keep this on the radar but at the moment I don't think this is going to be able to be added; if you'd like to take a crack at it, patches welcome!

Comment 4 Ayman 2019-10-26 14:14:09 UTC
So I've noticed that the swift build-script takes '--sourcekit-lsp [BOO]' and '--install-sourcekit-lsp [BOOL]' to build sourcekit-lsp along swift. I got swift to build fine on F30, but fails when building it with sourekit-lsp due to missing toolchain. When building a toolchain, sourcekit-lsp is included by default which means it is part of the release. Check out https://github.com/apple/sourcekit-lsp/releases/tag/swift-5.1.1-RELEASE
I'd say since it's part of the release, it should be really considered.

Comment 5 Ron Olson 2019-10-27 19:13:13 UTC
Okay, I'll give it a try.

Comment 6 Ron Olson 2019-10-31 17:38:46 UTC
I am able to include sourcekit-lsp into the package. However, I haven't had any success getting it to work with VSCode and am not exactly sure *how* it's supposed to work, which obviously isn't a great thing. Also, I posted a question on the Swift Forums (https://forums.swift.org/t/no-release-5-1-1-of-the-forked-llvm-project/30368) about there not being an explicit 5.1.1 release tarball of the llvm-project (not the same as llvm) and so I opted for a snapshot that was close to when Swift-5.1.1 was released. 

I have not submitted it to Koji because I'm not sure it's really working properly. If you want to test it, I am hosting the sourcekit-lsp version on Copr at https://copr.fedorainfracloud.org/coprs/tachoknight/swift-lang/ if you'd like to give it a try.

Comment 7 Ayman 2019-11-01 14:14:27 UTC
When installing the sourcekit-lsp version, I keep getting a Failed dependencies error:

error: Failed dependencies:
        libFoundationXML.so()(64bit) is needed by swift-lang-5.1.1-0.4.20191004git4242edd.fc30.x86_64

Installing it with --nodeps but no success with sourcekit-lsp. I tried nvim with lsp and vscode, the lsp server works but returns nothing. The same issue when building sourcekit-lsp from their master branch.
I'll keep testing and report back if there's any updates.

Comment 8 Ron Olson 2019-11-04 17:00:07 UTC
Hey Ayman-

The missing library is my fault; it wasn't packaged. I have a new build for x64_64 at https://tachoknight.fedorapeople.org/swift-lang/ if you'd like to try it. 

That said, it seems Apple has done some project reorganization; certain sub-projects have been deprecated for new ones, etc. I'm going to "start over" by trying to adhere as close as possible to their new layout as ultimately 5.1.2 or whatever the next -RELEASE version is will likely require it. I'll include sourcekit-lsp into it. 

In the meantime, if you can test the versions above I'd love feedback. Also, some general pointers on how to use the thing would be helpful as currently it packages, but I don't really have any idea how to judge whether it works or how it works; getting the REPL working was easy by comparison, it either started properly or it didn't. :)

Comment 9 Ayman 2019-11-06 03:33:45 UTC
Hey Ron,

I really appreciate the work you're doing, thank you!

So I've tried the new build and it installs fine but doesn't come with sourcekit-lsp. Tried building sourcekit-lsp from source but the build fails in swift-package-manager. So I'm back to the packages from copr.

sourcekit-lsp is a language-server, it helps with writing code. The guys behind it provide a plugin to work with VSCode. Once you install it and configure it, you should be able to get code autocompletion, hints, etc. Now I've tried it on an Ubuntu instance on a VM with swift-5.1-RELEASE-ubuntu18.04 toolchain and it worked out of the box! I just had to install the plugin and set the sourcekit-lsp binary path in VSCode.
Below are two pictures showing how it works on Ubuntu and how the lsp logs look like on Fedora after setting tracing to verbose.


Comment 10 Ron Olson 2019-11-06 16:20:30 UTC
The big difference I've noticed between the Apple Ubuntu builds and mine is that they pull straight from source on GitHub, whilst I'm using their tgz archives. The benefit of the former is that all the various projects are pulled down, including sourcekit-lsp as well as swift-format and swift-syntax. swift-format has no releases (i.e. no archive files to download), and swift-syntax is not in sync with the latest dev builds. To illustrate the differences in building: when you start Swift's REPL or do a `swift --version` on Ubuntu, you'll see that both swift and clang have git hashes, while the Fedora ones do not; this is because part of the building decorates the banners with the hash from the .git directory, while the archive files don't have the .git directory. 

So I thought about trying to copy what Apple is doing but with Fedora 31 and newer, which uses Python 3, that isn't possible because of differences in how 2 and 3 handle certain things. I have a pull request (https://github.com/apple/swift/pull/26296) which makes it all work under Fedora (and some of which I patch when building the RPMs), but Apple has not indicated anywhere I've found (mostly looking in the Swift forums) about when they're going to make a push to make Swift fully Python 3-compliant. 

TL;DR: I can't build everything that sourcekit-lsp needs because some sub-projects don't have releases, and I can't build everything from source because of Python versioning issues. 

I am updating to the 2019-11-05 snapshot, and will be making a COPR build as soon as my underpowered server finishes my test build (takes about four hours); I'll keep at it but wanted to let you know what's going on in case you don't see any meaningful progress. 

That said, I do totally want to have this feature now myself, so I'll definitely keep at it!

Comment 11 John Stalbaum 2019-11-11 08:42:55 UTC
Your package spec helped me get swift-lang built properly, thanks for this. I have some info to report on the off chance it'll be helpful.

While I was playing around, I noticed that your most recent package spec does not produce a working `sourcekit-lsp`. I used VSCode and `vim-lsp`, and although no errors were given, no completions or language server functionality was either.

Using your package spec to build a .tar.gz file and then installing the dependencies, followed by extracting this to somewhere on my hard drive and setting `PATH` (without the package installed), worked. e.g.

tar -xzf swift-5.1.2-fedora.tar.gz
mkdir -p /opt/swift
mv usr/ /opt/swift/usr
export PATH=$PATH:/opt/swift/usr/bin

I tested that by using my Vim plugin and VSCode. In that case, `sourcekit-lsp` produced completions against the standard library, member variables, etc...

So without digging any further, I would guess that something is up with the relative file structure that's causing the problem.

Finally, when I was building the image on an empty Docker image, I noticed that despite installing all of the dependencies in the spec file with `yum build-dep`, `make` was not installed and the build failed the first time. There's a good chance this is due to my ignorance (I'm relatively new to using the Fedora distribution) but I figured I'd include it here in the off chance it's significant.

Comment 12 Ron Olson 2019-11-11 16:35:50 UTC
Hi John-

So we're on the same page, the swift-5.1.2-fedora.tar.gz file you used came from building Swift via `rpmbuild` right (i.e. you were building it from the spec file)?

Also make is not necessary because Swift uses cmake to build.

Comment 13 John Stalbaum 2019-11-11 20:40:38 UTC
> So we're on the same page, the swift-5.1.2-fedora.tar.gz file you used came from building Swift via `rpmbuild` right (i.e. you were building it from the spec file)?

My procedure was roughly:
docker run -it fedora:31 bash
dnf install fedora-packager fedora-review
cd ~/
mkdir packaging
cd packaging
# probably missing some steps here
fedpkg --release f31 local

The build failed for the above reason (make was missing) so I installed it and continued the build by running

cd swift-build
swift/utils/build-script --preset=buildbot_linux,no_test install_destdir=$PWD/build installable_package=$PWD/build/swift-5.1.2-fedora.tar.gz

But the tar file that came out would probably look the same after the build completed successfully anyway.

I tested your changes by running:

sudo dnf copr enable tachoknight/swift-lang
sudo dnf update swift-lang

> Also make is not necessary because Swift uses cmake to build.
Doesn't CMake just create build files for different downstream build systems? So when "make" is run it's calling into `Makefile`s that have been generated by CMake, but still use `make` to execute? So wouldn't a CMake project that has been processed with the "Unix Makefile" generator still require that make be installed for the build to run? It's possible that this dependency problem has been masked by usually doing the build on machines that have make installed (as most development machines do).

I can try to reproduce this later if you'd like. Sorry if my report here is overly vague, but I don't have my terminal history available.

Comment 14 Ron Olson 2019-11-11 21:10:21 UTC
Right, I said CMake, but I really meant Ninja; you're correct that CMake generates files for downstream builders, which in this case is ninja.

Oh, so you built Swift from scratch, from GitHub? I do that too except it fails 100% of the time because they haven't merged my pull requests to make everything python 3 compatible.

I would, if you be so kind, to know what steps you actually did so I can compare. One thing that is always going to be different is that if you build from source without patching, you'll get a working swift installation that assumes everything is in known locations, for example the swift binary expects to find lldb in the same directory (usr/bin) and for packaging, that has to be placed elsewhere (/usr/libexec/swift-lldb in this case). 

My concern is that the sourcekit binary is doing something similar; I haven't been able to actually get sourcekit working to validate what's going on, but I'll keep at it.

Comment 15 John Stalbaum 2019-11-11 21:14:57 UTC
I was using a Python virtual environment to get around the Python problem.

However, this just underscores why I should put together a set of repro steps. :) I'll get to that this evening or tomorrow. Thanks for your time.

Comment 16 Ayman 2019-12-03 14:19:06 UTC
Any updates?

Comment 17 Ron Olson 2019-12-03 15:54:00 UTC
Was hoping to get some info from John; I've been working on it in the background when Real Life™ isn't getting in the way, which it has been lately. Also, when Apple stops releasing "swift-5.1-DEVELOPMENT-SNAPSHOT-" versions, it typically means they're doing big changes or at least changes that will break my build scripts so will have to figure out what's going on there. 

Anyway, TL;DR: not much yet other than coming up with new ways to break stuff. :)

Comment 18 Ron Olson 2019-12-04 22:47:42 UTC
just as an additional bit of info, my current process has me learning exactly how the vscode plugin works and communicates with clangd; not being much of a javascript/npm user this has been a bit of a journey, but work is, in fact proceeding, if not always in a completely straight direction. If anyone knows more about the integration and can help short-circuit this, I'd appreciate the info!

Comment 19 John Stalbaum 2019-12-06 07:18:56 UTC
Hey Ron & Ayman,

Sorry for disappearing. That Real Life guy is relentless.

Here are my repro steps:

sudo docker run -it fedora:31 bash
dnf -y install fedora-packager fedora-review which make
mkdir -p ~/packaging
cd ~/packaging
git clone https://src.fedoraproject.org/rpms/swift-lang
git reset --hard d968601519670f3fee5b06a03dc0cd2173ee00b4
cd swift-lang
dnf -y builddep ./swift-lang.spec
fedpkg --release f31 local
dnf -y install ~/packaging/x86_64/swift-lang-5.1.2-0.1.20191107git71def56.fc31.x86_64.rpm
# To test sourcekit-lsp
dnf -y install neovim
cd ~
git clone https://github.com/indianajohn/dotfiles/
git checkout swift
ln -s $HOME/dotfiles/.config $HOME/.config
curl -fLo ~/.local/share/nvim/site/autoload/plug.vim --create-dirs \
nvim +'PlugInstall --sync' +qall
nvim hello.swift                               
# Type print("hello")                                                                              
# No standard library completions

dnf rm swift-lang
cd ~/packaging/swift-lang
mkdir -p /usr/local/opt/swift
tar -xzf ~/packaging/swift-lang/swift-5.1.2-fedora.tar.gz -C /usr/local/opt/swift                                                                           
export PATH=$PATH:/usr/local/opt/swift/usr/bin
nvim hello.swift
# Standard library completions work

Standard library completions look like, for example:
pri use of unresolved identifier 'pri'
~ print(items: Any...)                                                               function Void    
~ print(items: Any..., separator: String, terminator: String)                        function Void    
~ print(items: Any..., separator: String, terminator: String, to: &TextOutputStream) function Void    
~ print(items: Any..., to: &TextOutputStream)                                        function Void    
~ private                                                                            keyword       

After typing "pri" in insert mode in Vim. The VSCode plugin likely uses the same mechanism, so the repro steps will be siimilar, as will the failure mode.

Also, more clarity on why `make` is needed; if not, then in this minimal Docker environment the following error pops up:
ICU for C/C++ 61.2 is ready to be built.
=== Important Notes: ===
Data Packaging: library
 This means: ICU data will be linked with ICU. A static data library will be built.  A shared data library will be built.
 To locate data: ICU will use the linked data library. If linked with the stub library located in stubdata/, the application can use 
udata_setCommonData() or set a data path to override.
** WARNING: make may not be GNU make.
This may cause ICU to fail to build. Please make sure that GNU make
is in your PATH so that the configure script can detect its location.
checking the version of "make"... /root/packaging/swift-lang/swift-source/icu/icu4c/source//configure: line 9081: make: command not found

Hope this is helpful.

Best Regards,

Comment 20 John Stalbaum 2019-12-06 07:21:37 UTC
Sorry, missed a `cd dotfiles` under `git clone https://github.com/indianajohn/dotfiles/`.

Comment 21 Ayman 2019-12-06 17:22:56 UTC
Hi John,

Can you share your swift-5.1.2-fedora.tar.gz?

Comment 22 Ron Olson 2019-12-06 22:28:44 UTC
Okay, I think I've found the problem. Gonna try something tonight, but because it takes a full four+ hours to compile everything (sometimes longer :( ), it'll probably be sometime tomorrow night at the earliest when I can update this ticket with what I've found.

The working theory is that I'm not including the version of clangd that is built as part of the toolchain, but instead am relying on the one from the clang-tools-extra package. I think this version of clangd doesn't understand swift, similarly to the way the packaged version of lldb also does not understand Swift and that's why I had to put it in /usr/libexec/swift-lldb so that it could live on the system alongside the regular one.

The plan is to move the Swift version of clangd to the same directory (/usr/libexec/swift-lldb) and modify the source on sourcekit-lsp via a patch to look for clangd in that directory, instead of the default bin directory. The file in question to be changed is source kit-lsp/Sources/SKCore/Toolchain.swift in case anyone at home is keeping score.[1] 

Once I get that all worked out and rebuilt I'll see how it goes with neovim per John's info; I was able to get sourcekit working on neovim using an Ubuntu-based system and it worked fine so I have a baseline to compare against.

[1] American baseball reference :)

Comment 23 Ron Olson 2019-12-08 18:38:44 UTC
Created attachment 1643094 [details]
screenshot of working sourcekit-lsp on fedora

Good news, it works! I built a test version with my patch and installed neovim and am able to get code-completion (the neovim suggestion is great because that I can test from the terminal).

BTW, everything that I use to build Swift for Fedora is here: https://github.com/tachoknight/swift-lang-packaging-fedora. That's my raw, working repo, unlike the official Fedora one that doesn't include the shell scripts, python code, etc. 

I have a copr repo that I can make a build for testing, or I can just push it to the official repo and hopefully folks can test it before it goes live. :)

Comment 24 Ron Olson 2019-12-09 00:03:03 UTC
FYI, if you want to try it *now*, the RPMs are available at https://tachoknight.fedorapeople.org/swift-lang/. Those are the ones I used.
Also, on my Fedora workstation, I installed it and sure enough, Visual Studio Code works great with sourcekit-lsp. 

One thing that probably needs to be considered is that installing the VSCode plugin isn't part of the installation process of Swift; the user would still have to go through the process of adding it to VSCode because there's no guarantee they have it installed or even want it, regardless of the fact they're installing Swift. Regardless, I think there's some rule about not automatically installing plugins to other apps; I may be wrong about this so I'll check.

Comment 25 John Stalbaum 2019-12-09 05:59:18 UTC
FYI I just tested this on my machine and it works as expected. Thanks for looking into it.

Comment 26 Ayman 2019-12-09 18:06:11 UTC
Tested here, works as expected. Thanks for your work!

Comment 27 Ron Olson 2019-12-11 03:10:05 UTC
FYI I can't deploy a new version because there seems to be a weird Glibc bug in rawhide that is preventing it from building correctly. I am going to submit a bug report against the Glibc project and we'll see what happens, but as it is I'd suggest sticking with the version downloadable from the link above for the time being.

The next Fedora release I can do will close this ticket so you'll know when it's ready. :)

Comment 28 Fedora Update System 2019-12-14 19:17:33 UTC
FEDORA-2019-8316fb625b has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-8316fb625b

Comment 29 Fedora Update System 2019-12-14 19:19:04 UTC
FEDORA-2019-c5c1f95c38 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-c5c1f95c38

Comment 30 Fedora Update System 2019-12-14 23:06:02 UTC
FEDORA-EPEL-2019-603d479a37 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-603d479a37

Comment 31 Fedora Update System 2019-12-15 01:28:00 UTC
swift-lang-5.1.3-0.1.20191213git005fc1f.el8 has been pushed to the Fedora EPEL 8 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-603d479a37

Comment 32 Fedora Update System 2019-12-15 01:41:03 UTC
swift-lang-5.1.3-0.1.20191213git005fc1f.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-8316fb625b

Comment 33 Fedora Update System 2019-12-15 01:48:33 UTC
swift-lang-5.1.3-0.1.20191213git005fc1f.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-c5c1f95c38

Comment 34 Fedora Update System 2020-01-04 20:07:48 UTC
swift-lang-5.1.3-0.1.20191213git005fc1f.el8 has been pushed to the Fedora EPEL 8 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.