Improving Addon Installation #2408
Replies: 7 comments 34 replies
-
|
Without any expertise in this area, the feature that seems most critical to me is the ability to install an addon from a url. The json registry features you mention would be a nice convenience on top of that, but they're not something I'm personally attached to. The registry could be as simple as a web page with a list of links to addon repos for popular frameworks and I wouldn't complain. It also seems like if the json registry were to be set up, it would be built on the "download from url" tool, so I would think any immediate efforts should start there. I don't know the details of how lls hooks into editors, so I don't know how enabling/disabling certain addons for certain contexts could work in a way that does not involve settings files specific to the current editor, but I'd imagine that's a feature people would need? Given that addons have unique names it could be something that happens within the script, e.g. at the top of a script you could have something along the lines of "---@require busted". With a registry setup that could be pushed as far as automatically scanning the registry for the requested addon and installing it, although that's probably overkill. Alright that's all the thoughts I have, cool project =) |
Beta Was this translation helpful? Give feedback.
-
|
You're correct, the first step is definitely just getting installing from a URL working. The registry is meant to serve two purposes:
The perceived value added by purpose number 1 may differ greatly from person to person. It is possible to even have that convenience without the registry by just looking in the LuaCATS organization for a matching addon name. However, purpose number 2 is much more valuable in my opinion. It allows people to publish their addon, give it a nice display name, and a description. Anyone can then use this JSON file for listing all the published addons, for example, the addon manager UI could use this. The language server's settings are the same across editors. The only "special case" here is that VS Code can have additional VS Code specific settings applied using the Looking for addons in Thanks for the ideas! I had never considered not doing the registry at all and instead using the LuaCATS organization as the "registry". While I will probably still do the JSON registry, it is an interesting idea. |
Beta Was this translation helpful? Give feedback.
-
|
The registry would definitely be an upgrade if you can get it to work! |
Beta Was this translation helpful? Give feedback.
-
|
Has there been any progress on this aspect? Hardcoding plugin paths in Maybe one way to improve it would be to allow URLs in "Lua.workspace.library": [ "github:my/plugin" ] // gets the latest commit from https://github.com/my/plugin
"Lua.workspace.library": [ "github:my/plugin#dc4c9f2" ] // like above, but pinned to commit `dc4c9f2`You can also specify other URL types like: "gitlab:my/plugin" // resolves to https://gitlab.com/my/plugin
"git:https://fictional.git.forge/my/plugin" // a custom git forge
"url:https://example.com/plugin.lua" // direct URL download |
Beta Was this translation helpful? Give feedback.
-
|
You can use I have been looking at making a custom builder for LuaRocks that would be able to install an addon using only LuaRocks. I have been able to get it to download the rock and move it to the right directory. There is lots of work remaining in order to get it to apply the addon configuration to the right config file, though, and I do not currently have time to work on it. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, |
Beta Was this translation helpful? Give feedback.
-
|
What you are trying to achieve is a command-line-based plugin management tool. In that case, there's no need to use LuaLS itself. You can use any programming language you prefer to implement such a tool and then configure LuaLS to recognize the location of the plugins based on a configuration file like ".luarc". |
Beta Was this translation helpful? Give feedback.
-
|
If we consider git as the command-line tool, what we need to do is set up a central index repository to index different versions of plugins. This central index repository would periodically release a release, and LuaLS can pull a plugin list from that release with each release it publishes. This way, when requiring a plugin, it can prompt for third-party plugins and provide an option to automatically download them (using git). |
Beta Was this translation helpful? Give feedback.
-
|
You have just described the current method almost exactly 😄. LLS-Addons is the "central index" and the VS Code extension clones it and enables specific addons. Creating a completely separate CLI tool is an option, but like I mention above, it would be best if it could be implemented inside LuaLS:
|
Beta Was this translation helpful? Give feedback.
-
|
Actually, I just thought of an alternative to the "official" JSON registry idea. I can instead use the GitHub/GitLab API to search for repositories that have a certain topic. This could allow us to simply ask GitHub/GitLab for a list of all projects that have the Using this instead means there is no quality control and reviewing of addons, but it does mean there is no need to maintain an "official" list. We could of course just have this as a fallback should the addon name provided not exist in the official list. |
Beta Was this translation helpful? Give feedback.
-
|
What about using LuaRocks for installing/uninstalling addons? Devs can create a Creating an AddonDefinitions can live in a -- example-types-1.2.1-1.rockspec
name = "example-types"
version = "1.2.1-1"
description = {
summary = "types for Example 1.2.1",
detailed = [[
This is an example of how types would be specified and published via LuaRocks.
]],
homepage = "https://github.com/goldenstein64/example-types",
license = "MIT",
labels = { "types" },
maintainer = "goldenstein64 <email>"
}
source = {
url = "https://github.com/goldenstein64/example-types"
}
dependencies = {
"lua >= 5.1",
-- other type definitions this module depends on
"other-types ~> 0.8.6",
}
build = {
type = "none",
-- this creates a folder at lua_modules/lib/luarocks/rocks-5.1/example-types/1.2.1/types
copy_directories = { "types" }
}The language server can also create extra tooling for help with writing this file, maybe by executing Publishing an AddonDevelopers can publish their addon as a module using the LuaRocks workflow, either through the website or via CLI, i.e. Manifests are described on the LuaRocks site as just a collection of modules. They are simple to create, and manifest maintainers have the choice of either manually reviewing included modules or marking the manifest as open, letting anyone contribute. Installing an AddonUsers can install an addon using the LuaRocks workflow using their CLI, i.e. Users can also browse addons on the LuaRocks website through the public manifest, i.e. https://luarocks.org/m/lls-addons. For lesser-used addons, they don't have to be part of the |
Beta Was this translation helpful? Give feedback.
-
This has been an issue with the LuaRocks site for a while with not much discussion, e.g. luarocks/luarocks-site#59 and luarocks/luarocks-site#141.
Not much to add here, but maybe you could use a parser over a full-blown VM? That probably wouldn't work well because even rockspecs expect Lua semantics to "build" the globals in a rockspec. |
Beta Was this translation helpful? Give feedback.
-
|
Seeing as that issue 59 has been around since 2015... I think getting access to a custom manifest's JSON file is off the table. Add on top that the rockspec files are written in Lua, and now using Luarocks is perhaps not possible. |
Beta Was this translation helpful? Give feedback.
-
My PR for serving non-root manifests (luarocks/luarocks-site#209) was just merged! That should hopefully clear any concerns about trouble with accessing non-root manifests. I say "hopefully" because I haven't seen the changes pushed to the site as of this comment. About the lack of HTTPS support, it may be helpful to know that luarocks.org also supports retrieving manifests and rockspecs through plain HTTP. Given that whatever is on the registry is already public information, security may not be a big concern when strictly using it for retrieval? If a makeshift HTTP API was made, it should be restricted to GET requests and forbid cookies or something.
This is still a blocker, but some good news would be that rockspecs are interpreted without globals. So if a Lua VM had to be set up, you wouldn't need the standard library, metatables, threads, userdatas, etc. If there is a way to remove those parts from the VM, maybe that would make it lean enough to embed into the addon manager. |
Beta Was this translation helpful? Give feedback.
-
That's awesome! Hopefully we will see the changes go live soon, that would be a big step in the right direction. This might just get me back into looking at LuaRocks as the package manager for the addons 😉.
I've been thinking the addon manager really wouldn't belong embedded in the language server. They are really two related yet separate things. Some people may want the language server without the addon manager. I would probably just write a pretty simple CLI tool using Go that downloads the addons from LuaRocks and then sets up your workspace settings. Another advantage of this would be that I have been working on and off on a Lua parser in Go that should at least be able to deal with the simple rockspecs - could give me a reason to finish it 😄. |
Beta Was this translation helpful? Give feedback.
-
The merge went live just now! 🎉 |
Beta Was this translation helpful? Give feedback.
-
there is build_dependencies which works |
Beta Was this translation helpful? Give feedback.
-
|
I created a simple GitHub-based lua package management tool, which might initially address this need. https://github.com/CppCXY/luap |
Beta Was this translation helpful? Give feedback.
-
|
This appears to be fairly similar to the current addon manager in the VS Code client, as it uses Git to install packages — without the use of LLS-Addons. Using git has some positives, as explained in the original post, however, it also has some negatives like copying the entire directory structure. For many addons, it makes sense to have a “build” step where the addon is generated, and an output is prepared with the minimum files necessary. That is why I want to also support checking GitHub releases and GitLab releases. This would allow maintainers to have their action runner publish their addon with whatever files they choose. This is sort of, roughly, similar to how Go's package management works where they first check their proxy for the package before defaulting to grabbing the package straight from GitHub/GitLab/etc. via Git. Ideally, we could also just refer to packages by their URL, like This is also a very slippery slope where you start by creating a simple package manager for addons for the language server, and very quickly start re-inventing LuaRocks with building C, a dedicated package registry, etc. 😆 |
Beta Was this translation helpful? Give feedback.
-
Actually, I didn't use Git directly; I used libgit2. It's also easy to download from releases use rust. Indeed, implementing a complete Lua package manager is like reinventing LuaRocks. It's not very difficult, but I don't have the energy to focus on these things, so I only implemented the package that can simply download docs for annotation. If anyone wants to further extend this project, they are very welcome to do so. |
Beta Was this translation helpful? Give feedback.
-
|
I made a Rust project that wraps LuaRocks in another CLI for installing LuaLS addons called llynx: https://github.com/goldenstein64/llynx It depends on a LuaRocks installation, which in turn depends on a Lua installation. It assumes that:
All these assumptions are configurable with CLI arguments, which can be saved in a TOML config file. It defaults to looking in a It's still in its early stages, but if this design is acceptable, maybe it could be used as a de-facto addon manager. |
Beta Was this translation helpful? Give feedback.
-
|
When I was writing that, I was definitely thinking of just the types; I forgot that they are also able to modify the settings file! It also doesn't have any support for enabling a plugin. I think it would be possible to implement both features in llynx. A custom backend for installing addons would likely lead to a much better developer experience than installing another executable. It's a creative solution that I never thought about. If I were to go down that route, I would:
Example code-- src/luarocks/build/lls-addon
local jsonc = require("jsonc") -- not sure where this would come from
local fs = require("luarocks.fs")
local lls_addon = {}
local function deepEqual(a, b)
if type(a) == "table" and type(b) == "table" then
for k, v in pairs(a) do
if b[k] ~= v then
return false
end
end
return true
else
return a == b
end
end
local function contains(array, value)
for _, v in ipairs(array) do
if deepEqual(v, value) then
return true
end
end
return false
end
local function extend(old, new)
if type(old) == "table" and type(new) == "table" then
if #old > 0 and #new > 0 then -- treat arrays like sets
for _, v in ipairs(new) do
if not contains(old, v) then
table.insert(old, v)
end
end
else
for k, v in pairs(new) do
old[k] = extend(old[k], v)
end
end
return old
else
return new
end
end
function lls_addon.run(rockspec, no_install)
assert(rockspec:type() == "rockspec")
local SETTINGS_PATH = os.getenv("LLS_SETTINGS_PATH")
local lib_rock_path = -- there must be a way to get this, right?
local settings
do
local settings_file = assert(io.open(SETTINGS_PATH, "r"))
local settings_contents = settings_file:read("a")
settings_file:close()
settings = jsonc.decode(settings_contents)
end
-- add types if they exist
local library_path = lib_rock_path .. "/types/library"
if fs.exists(library_path) then
table.insert(settings["Lua.workspace.library"], library_path)
end
-- add the plugin if it exists
local plugin_path = lib_rock_path .. "/types/plugin.lua"
if fs.exists(plugin_path) then
settings["Lua.runtime.plugin"] = plugin_path
end
-- merge the rockspec settings
if rockspec.build.settings then
extend(settings, rockspec.build.settings)
end
do
local settings_contents = jsonc.encode(settings)
local settings_file = assert(io.open(SETTINGS_PATH, "w"))
settings_file:write(settings_contents)
settings_file:close()
end
end
return lls_addon |
Beta Was this translation helpful? Give feedback.
-
|
Luarocks is poorly made and not worth using directly. I suggest you use https://github.com/nvim-neorocks/lux instead. |
Beta Was this translation helpful? Give feedback.
-
|
The build backend solution was really just a creative workaround to make LuaRocks itself work for this. It is definitely not ideal. The hosting of addons on LuaRocks and the backend itself is fine, it is just a bit clunky to require a project that just wants types to depend on an addon as a dependency, but I don't think there is a "dev dependency" like with NPM. I was able to recover some of the code using a disk recovery tool and get it at working in a very basic fashion. I have just invited you to it, @goldenstein64. All it is capable of right now is downloading an addon from LuaRocks and copying the |
Beta Was this translation helpful? Give feedback.
-
@CppCXY I've heard of Lux from other communication channels; I don't want to build on top of something that doesn't have a stable release. If you think it won't change anytime soon, I'll consider it. And besides that, Lux supports all of LuaRocks' build features by falling back to LuaRocks commands. |
Beta Was this translation helpful? Give feedback.
-
|
I agree with @goldenstein64, Lux seems promising for the future, but I don't think it would be wise to build upon something that does not yet have a stable release. Hisham has sadly, but understandably, been discussing stepping down from maintaining LuaRocks, but for now, everyone is still using LuaRocks. Besides, as far as I can tell, Lux uses all of LuaRocks' backend and is more of an alternate CLI rather than a completely new non-backwards-compatible package ecosystem. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I have for a long time wanted to get this done, so let's get the ball rolling. Please feel free to provide feedback!
Current Implementation
Currently, if you use VS Code, you can use the addon manager, a webview UI that uses Vue.js to display the available addons from LLS-Addons.
More technically, the UI communicates with the VS Code extension through message ports, as the UI is in an
<iframe>context. The UI does not handle the installing/enabling of addons, it simply sends a message to the extension to install/enable a specified addon.The way that the addons are installed by the VS Code extension, is by using Git to clone LLS-Addons and then the specific addon submodule is initialized. This has a few positives and negatives:
Positives
Negatives
New Implementation Idea
Let me step through how I think we can improve this.
Creating an addon
This step is very similar. You can have definition files in a
library/folder, and you can have a plugin. Every addon can optionally contain asettings.jsonfile, that defines settings that the language server should apply (this is the same info as thesettingsfield in the currentconfig.json. Every addon is required to have anaddon.jsonfile that contains info about that addon, like the below:{ "version": "1.0.0", "minServerVersion": "4.0.0", "name": "example", "author": "carsakiller", "description": "An example for demonstrating a basic use case for a lua language server addon.", "repository": "https://github.com/carsakiller/lls-addon-example" }Publishing an addon
LLS-Addons should be replaced with a new "addon registry" repo that contains an easy to parse JSON file of "officially" published addons. The JSON could look like this:
{ "example": { "name": "Example", "author": "carsakiller", "description": "An example for demonstrating a basic use case for a lua language server addon.", "repository": "https://github.com/carsakiller/lls-addon-example" } }Firstly, you use GitHub actions or GitLab CI to create a release that contains your addon as a zip file. Then, once you have a public release available for download, you open a pull request that adds the above info about your addon to the JSON file. Any addon-installing client can then fetch the raw JSON file and have all the officially published addons.
Installing an addon
This is where the bulk of the work is.
While using Git is very convenient, we are not using it as intended and it can behave unexpectedly. Therefor, I think we should switch to simply using the GitHub and GitLab APIs. In the future, we could also add any other platforms that have an API.
The logic for installing addons should move to the language server, that way, wherever it goes, whichever editor someone is using, they are able to install and enable addons. It could instead become its own CLI tool, which should be pretty easy, but then users have to install a completely separate tool, and we will be duplicating the logic that enables addon settings 🙁. To integrate it into the language server, I will need help @sumneko with getting comfortable with the runtime and tooling. I have many questions, here are just a few:
$ lua-language-server addon enable busted --plugin)zipInstall Flow
So let's say you want to enable the
bustedaddon, which has been published to the official registry:busted. Assuming there is a match, we can look at therepositoryfield to know where it lives.workspace.librarysetting and any additional settings (according to itssettings.json)addon.jsonfile, and after it is installed, any dependencies can be resolved and enabled.But what if you have an addon for a small project that isn't published to the official registry? Well, we can also support simply providing the URL to it to install it:
$ lua-language-server addon enable https://github.com/example/addonWe can then use the GitHub API to get the latest release and then follow the same steps as above.
Addon Manager UI
Of course, these changes will completely break the addon manager UI that is being used by the VS Code extension. The UI itself may be easy to fix, as it is just passing messages, but the code that handles those messages and runs git commands will have to all be torn out and refactored. We could maybe add a command like
$ lua-language-server addon list --jsonto get the addon registry JSON file. The UI could then continue displaying the official addon list and interact with the language server binary to enable addons.Please let me know what you think of these ideas, and feel free to offer your own suggestions!
Beta Was this translation helpful? Give feedback.
All reactions