Add lua_eval command, lua IPC command, and scroll.context_container API

[jbms]

lua_eval allows inline lua to be evaluated without needing to write it to a file.

Compared to just running lua or lua_eval via RUN_COMMAND, the new LUA_EXEC command allows a return value to be sent back to the client.

The scroll.context_container() API provides access to the context container when lua or lua_eval is run with criteria.

[dawsers]

In a194801 I added a Lua REPL to scrollmsg, and an IPC command, LUA_EVAL. I don't know how it will affect this, and iwhetherit is enough for what you are trying to do.

[jbms]

It is enough for what I need for the session manager program. However I do have a few comments about that commit:

• A linux pipe has a fixed-size buffer and writes to it will hang once that buffer is filled. Therefore if too much gets written to redirected stdout, the compositor will hang. To solve, I think there are a few options: (1) have another thread reading from the pipe (kind of unfortunate to introduce another thread just for this purpose) (2) implement the redirection using lua APIs rather than actually redirecting stdout (3) set the pipe to non-blocking mode so that writes will fail if the buffer gets full --- it would be good to check how lua handles this error. You could also call fcntl F_SETPIPE_SZ to increase the buffer size in combination with that.
• It would be nice to avoid having to duplicate the names of all of the lua functions again for use by the repl --- maybe it can instead be queried using the lua API?
• It would still be nice to provide a lua_eval command so that you can e.g. bind a key to run a lua expression directly without needing a separate file. In particular this would allow defining a lua function as a global variable and then calling it from a key binding rather than having to load it from a file for each invocation.
• Although not necessary for my session manager, since it uses the IPC API directly rather than via scrollmsg, it would be nice to expose lua_eval as a regular IPC command in addition to the lua repl.
• Although not strictly necessary it might be nice to allow arguments to be passed to the lua code as JSON values, since that would be consistent with the return value being returned as JSON and avoid the need to implement "encode json-like value as a lua expression" in whatever language is being used with the IPC API. json encoding is much more readily available than lua literal encoding.

If you are open to any of these changes I'm happy to update this PR to make those changes on top of current scroll HEAD.

[dawsers]

Thank you for your comments, they are all very insightful.

A linux pipe has a fixed-size buffer and writes to it will hang once that buffer is filled.

I should have been more thorough instead of supposing "nobody" would print more than 64 KB to stdout. In 7bca823 I set the reader in a different thread. It now supports any output size and also makes the code is cleaner.

It would be nice to avoid having to duplicate the names of all of the lua functions again for use by the repl --- maybe it can instead be queried using the lua API?

I also wanted to avoid this. The problem is scrollmsg is a different program and it doesn't have access to sway's library. The quick and dirty solution was what I did. An even dirtier one would have been to create a common lua.h header and put all the function names there, but note that currently, the Lua registry for scroll lib is not visible, it is a static array in sway/lua.c. So until there is a bigger refactor, I'd rather leave it as it is. The worst that can happen is I forget to add some function to the auto-complete for a while, but it is just a tool to help, not something critical.

It would still be nice to provide a lua_eval command so that you can e.g. bind a key to run a lua expression directly without needing a separate file. In particular this would allow defining a lua function as a global variable and then calling it from a key binding rather than having to load it from a file for each invocation.

I could do that, I just think the config file would become a mess if people abuse this, and then file issues here because "something doesn't work", leaving me to debug huge config files that also include Lua code. Scripts also allow using arguments and the state, so this facility would be a bit limited, probably only useful to run a couple of commands that can be run without Lua anyway. If you have something specific and useful in mind, please share.

Although not necessary for my session manager, since it uses the IPC API directly rather than via scrollmsg, it would be nice to expose lua_eval as a regular IPC command in addition to the lua repl.

I added this in cf9c88d. Now you can do `scrollmsg -t lua_eval 'print("Hello")'.

lthough not strictly necessary it might be nice to allow arguments to be passed to the lua code as JSON values, since that would be consistent with the return value being returned as JSON and avoid the need to implement "encode json-like value as a lua expression" in whatever language is being used with the IPC API. json encoding is much more readily available than lua literal encoding.

What do you mean? For scripts? Or you want to share data from Python? How would someone use this so I understand better?

[jbms]

Thank you for your comments, they are all very insightful.

A linux pipe has a fixed-size buffer and writes to it will hang once that buffer is filled.

I should have been more thorough instead of supposing "nobody" would print more than 64 KB to stdout. In 7bca823 I set the reader in a different thread. It now supports any output size and also makes the code is cleaner.

It would be nice to avoid having to duplicate the names of all of the lua functions again for use by the repl --- maybe it can instead be queried using the lua API?

I also wanted to avoid this. The problem is scrollmsg is a different program and it doesn't have access to sway's library. The quick and dirty solution was what I did. An even dirtier one would have been to create a common lua.h header and put all the function names there, but note that currently, the Lua registry for scroll lib is not visible, it is a static array in sway/lua.c. So until there is a bigger refactor, I'd rather leave it as it is. The worst that can happen is I forget to add some function to the auto-complete for a while, but it is just a tool to help, not something critical.

Maybe the repl can just use the LUA_EVAL command on startup or on first completion request to obtain the members.

It would still be nice to provide a lua_eval command so that you can e.g. bind a key to run a lua expression directly without needing a separate file. In particular this would allow defining a lua function as a global variable and then calling it from a key binding rather than having to load it from a file for each invocation.

I could do that, I just think the config file would become a mess if people abuse this, and then file issues here because "something doesn't work", leaving me to debug huge config files that also include Lua code. Scripts also allow using arguments and the state, so this facility would be a bit limited, probably only useful to run a couple of commands that can be run without Lua anyway. If you have something specific and useful in mind, please share.

In general it just makes it easier to do things without having to write extra files. Having the indirection of a file just to invoke some lua function makes it more complicated, in my opinion, and forcing the code to go through a separate file rather than inline doesn't reduce the complexity, it just adds an extra indirection.

One example is that you could define a global function in a file (or inline lua code) that gets evaluated, and then define a key binding that just calls the function. Without this an external shim file is needed to do anything.

By the way I was not clear on the purpose of the state mechanism in the first place. You can anyway just access lua globals or use the _G table, which also allows sharing state between multiple scripts and with lua_eval. Your state mechanism provides a separate namespace for each lua file, but just using a suitable prefix for any global variables would also seem to be sufficient. I also wasn't clear why state isn't just exposed directly as a lua table.

Although not necessary for my session manager, since it uses the IPC API directly rather than via scrollmsg, it would be nice to expose lua_eval as a regular IPC command in addition to the lua repl.

I added this in cf9c88d. Now you can do `scrollmsg -t lua_eval 'print("Hello")'.

lthough not strictly necessary it might be nice to allow arguments to be passed to the lua code as JSON values, since that would be consistent with the return value being returned as JSON and avoid the need to implement "encode json-like value as a lua expression" in whatever language is being used with the IPC API. json encoding is much more readily available than lua literal encoding.

What do you mean? For scripts? Or you want to share data from Python? How would someone use this so I understand better?

For my session manager I want to use lua_eval to restore one or more windows as an atomic operation (similar to the spaces restore), which means I need to pass certain information about the ids and target position of the set of windows that will be affected. Being able to just encode this as JSON and then have scroll translate to lua is convenient so that I don't have to either add a json parser to lua, or a lua encoder to my program. I think in general it would be convenient for users to be able to pass structured data directly to their lua code without having to encode it as a lua expression.

[dawsers]

Maybe the repl can just use the LUA_EVAL command on startup or on first completion request to obtain the members.

Starting from 69e6e29, the scroll completions table is generated dynamically.

[dawsers]

For my session manager I want to use lua_eval to restore one or more windows as an atomic operation (similar to the spaces restore), which means I need to pass certain information about the ids and target position of the set of windows that will be affected. Being able to just encode this as JSON and then have scroll translate to lua is convenient so that I don't have to either add a json parser to lua, or a lua encoder to my program. I think in general it would be convenient for users to be able to pass structured data directly to their lua code without having to encode it as a lua expression.

Would this be covered if I added a function scroll.json_parse(string) that returns a Lua table after parsing the JSON data in string?

[jbms]

Yeah I suppose it would. Then I could encode it as json and then encode again as a json string, which I think would also be a valid lua string. Still given that the return value is encoded as json automatically it makes sense to me to offer a way to provide json values as input.

[dawsers]

Yeah I suppose it would. Then I could encode it as json and then encode again as a json string, which I think would also be a valid lua string. Still given that the return value is encoded as json automatically it makes sense to me to offer a way to provide json values as input.

I think I don't understand what you mean. I think you are saying, optimally you would want to input non-encoded JSON? How would you do that? I mean, what kind of interface would there be? JSON is portable in string format, so to be able to interface with other languages you need to convert from the library's internal json binary format to a JSON string, don't you? That is why I proposed local table = scroll.json_parse(json_data_in_string_format), to convert a JSON object in string format to a Lua table. Maybe you are thinking of something different, please explain.

[dawsers]

I am OK with the changes to sway/commands/lua.c, including cmd_lua_eval() if you reaaally want to have a way to call Lua from the config file. But lua_eval needs to assign a name to the "fake" script it creates, so there are no problems with the local state for the script. Currently, you assign "" as the name, so a second script will use the state of the previous one. The command would then be something like:

bindsym $mod+q lua_eval fake_name 'local args, state = ...\n for i = 1, 2, 1 do scroll.command(nil, args[1]) end' 'cycle_size h next'

[jbms]

Yeah I suppose it would. Then I could encode it as json and then encode again as a json string, which I think would also be a valid lua string. Still given that the return value is encoded as json automatically it makes sense to me to offer a way to provide json values as input.

I think I don't understand what you mean. I think you are saying, optimally you would want to input non-encoded JSON? How would you do that? I mean, what kind of interface would there be? JSON is portable in string format, so to be able to interface with other languages you need to convert from the library's internal json binary format to a JSON string, don't you? That is why I proposed local table = scroll.json_parse(json_data_in_string_format), to convert a JSON object in string format to a Lua table. Maybe you are thinking of something different, please explain.

Most of the other IPC commands use json for the input, so in my PR I used json for the lua ipc command and that naturally provided an easy way to specify additional arguments as json. Indeed they still need to be encoded as json though.

Either way I think it would be useful to provide json encode and decode functions in the lua api for user convenience if they exist in the c source code anyway.

[jbms]

I am OK with the changes to sway/commands/lua.c, including cmd_lua_eval() if you reaaally want to have a way to call Lua from the config file. But lua_eval needs to assign a name to the "fake" script it creates, so there are no problems with the local state for the script. Currently, you assign "" as the name, so a second script will use the state of the previous one. The command would then be something like:

bindsym $mod+q lua_eval fake_name 'local args, state = ...\n for i = 1, 2, 1 do scroll.command(nil, args[1]) end' 'cycle_size h next'

Maybe lua_eval just shouldn't get a state at all. Since users can just store state in global variables, and define global functions that reference file-local variables and then call those global functions from lua_eval, there are already a lot of options for state management that don't require the scroll state mechanism.

[dawsers]

Yeah I suppose it would. Then I could encode it as json and then encode again as a json string, which I think would also be a valid lua string. Still given that the return value is encoded as json automatically it makes sense to me to offer a way to provide json values as input.

I think I don't understand what you mean. I think you are saying, optimally you would want to input non-encoded JSON? How would you do that? I mean, what kind of interface would there be? JSON is portable in string format, so to be able to interface with other languages you need to convert from the library's internal json binary format to a JSON string, don't you? That is why I proposed local table = scroll.json_parse(json_data_in_string_format), to convert a JSON object in string format to a Lua table. Maybe you are thinking of something different, please explain.

Most of the other IPC commands use json for the input, so in my PR I used json for the lua ipc command and that naturally provided an easy way to specify additional arguments as json. Indeed they still need to be encoded as json though.

Either way I think it would be useful to provide json encode and decode functions in the lua api for user convenience if they exist in the c source code anyway.

My question was about how someone can use those functions from outside of scroll itself. I can provide them, but nobody can actually input "binary" JSON from a config file, scrollmsg, the IPC interface or a Lua script, they can only use standard JSON in text format.

In your code you use the functions in IPC_LUA_EXEC, to pass arguments that have previously been passed in text JSON format, parsed into a json_object, and then sent to Lua converting them again into a Lua table. That is no different than choosing for a script what format of arguments you want to use for the script, and if it is JSON, use an internal parser, local lua_table = scroll.json_parse(json_txt_formatted), and if it is not, there is no need to do all that parsing. That is what I meant.

[dawsers]

I am OK with the changes to sway/commands/lua.c, including cmd_lua_eval() if you reaaally want to have a way to call Lua from the config file. But lua_eval needs to assign a name to the "fake" script it creates, so there are no problems with the local state for the script. Currently, you assign "" as the name, so a second script will use the state of the previous one. The command would then be something like:

bindsym $mod+q lua_eval fake_name 'local args, state = ...\n for i = 1, 2, 1 do scroll.command(nil, args[1]) end' 'cycle_size h next'

Maybe lua_eval just shouldn't get a state at all. Since users can just store state in global variables, and define global functions that reference file-local variables and then call those global functions from lua_eval, there are already a lot of options for state management that don't require the scroll state mechanism.

Sure, but currently there is a state, and that is useful for people who don't want to have to deal with global stuff. If they want to, they can, but it not, scroll already provides a state that allows them to use "script-local" memory that stays there between runs. Being able to use the same scripts and be compatible with standard scroll scripts should be a goal when it is so easy to attain.

[jbms]

I am OK with the changes to sway/commands/lua.c, including cmd_lua_eval() if you reaaally want to have a way to call Lua from the config file. But lua_eval needs to assign a name to the "fake" script it creates, so there are no problems with the local state for the script. Currently, you assign "" as the name, so a second script will use the state of the previous one. The command would then be something like:

bindsym $mod+q lua_eval fake_name 'local args, state = ...\n for i = 1, 2, 1 do scroll.command(nil, args[1]) end' 'cycle_size h next'

Maybe lua_eval just shouldn't get a state at all. Since users can just store state in global variables, and define global functions that reference file-local variables and then call those global functions from lua_eval, there are already a lot of options for state management that don't require the scroll state mechanism.

Sure, but currently there is a state, and that is useful for people who don't want to have to deal with global stuff. If they want to, they can, but it not, scroll already provides a state that allows them to use "script-local" memory that stays there between runs. Being able to use the same scripts and be compatible with standard scroll scripts should be a goal when it is so easy to attain.

In my PR I just hard coded it to use the empty string as the state identifier because it is guaranteed not to conflict with any filename and because it seemed awkward to force users to specify a state identifier on every use even if they aren't actually using the state. The other issue with the state is that it isn't accessible via the lua_eval IPC command.

[dawsers]

I am OK with the changes to sway/commands/lua.c, including cmd_lua_eval() if you reaaally want to have a way to call Lua from the config file. But lua_eval needs to assign a name to the "fake" script it creates, so there are no problems with the local state for the script. Currently, you assign "" as the name, so a second script will use the state of the previous one. The command would then be something like:

bindsym $mod+q lua_eval fake_name 'local args, state = ...\n for i = 1, 2, 1 do scroll.command(nil, args[1]) end' 'cycle_size h next'

Maybe lua_eval just shouldn't get a state at all. Since users can just store state in global variables, and define global functions that reference file-local variables and then call those global functions from lua_eval, there are already a lot of options for state management that don't require the scroll state mechanism.

Sure, but currently there is a state, and that is useful for people who don't want to have to deal with global stuff. If they want to, they can, but it not, scroll already provides a state that allows them to use "script-local" memory that stays there between runs. Being able to use the same scripts and be compatible with standard scroll scripts should be a goal when it is so easy to attain.

In my PR I just hard coded it to use the empty string as the state identifier because it is guaranteed not to conflict with any filename and because it seemed awkward to force users to specify a state identifier on every use even if they aren't actually using the state. The other issue with the state is that it isn't accessible via the lua_eval IPC command.

Yes, but by using the empty string, you create a sway_lua_script with name "", and if then there is another call from another lua_eval for a different function/command/script, it will use the same state as the previous script.

The users wouldn't be specifying a "state identifier", they would be simply setting a name for the script, like any command or file would have one. It can be thought as something related to what the script does. And this way there is no state name clash.

Edit: I forgot to address this.

The Lua eval IPC command is to be able to interact with scroll in real-time, like the REPL, there is no need to arguments or state, everything goes in that particular interaction. That is why we are adding the lua_eval command, so there is a way to do things like in standard scroll, with state etc. And it can be called from scrollmsg too, like any other command. so we will have

scrollmsg -t lua_eval (IPC)

scrollmsg lua_eval ... (command).