Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure plugin start timeout for go-plugin.Client (#4477) #4478

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

nmische
Copy link
Contributor

@nmische nmische commented Jan 25, 2025

Addresses #4477.

The hashicorp/go-plugin client configuration has a default timeout of 60s. This causes steampipe startup to fail if a plugin takes longer than 60s.

This PR ensures a plugin's client is configured with the timeout settings from the steampipe plugin configuration.

It also extends the plugin manager client start timeout from 1 minute to 3 minutes as testing indicated this timeout setting also governed overall steampipe startup. Ideally this timeout would be set to the maximum plugin timeout setting, however I didn't see any easy way to achieve that so opted to extend the default for now.

@@ -64,6 +65,7 @@ func start(steampipeExecutablePath string) (*State, error) {
Cmd: pluginManagerCmd,
AllowedProtocols: []plugin.Protocol{plugin.ProtocolGRPC},
Logger: logger,
StartTimeout: 3 * time.Minute, //TODO: macke this configurable
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timeout for the plugin manager startup needs to be extended to accommodate longer plugin startup. I didn't see an immediate way to make this configurable, so I extended this from 1 minute to 3 minutes for now.

@pskrbasu
Copy link
Contributor

Thanks @nmische for the contribution. I will take a look at the changes and submit my review. 👍

@gkid
Copy link

gkid commented Feb 23, 2025

Bumping this - looks like it's causing me issues. Using the terraform plugin, If I put more than a few s3 paths to state files, especially if there are large files - I get a timeout error:

2025-02-23 11:24:33.401 UTC [WARN]  steampipe [9a40]: plugin manager start() failed to start GRPC client for plugin manager: timeout while waiting for plugin to start
2025-02-23 11:24:33.403 UTC [WARN]  steampipe [9a40]: StartServices plugin manager failed to start: failed to start plugin manager: timeout while waiting for plugin to start

I set plugin.start_timeout = 600 option in default.spc but it doesn't seem to have any effect

I'm hoping that this will fix the issue.

@ramses999
Copy link

I have 24 k8s cluster and I get the same error.

@pskrbasu
Copy link
Contributor

@gkid @ramses999 @nmische
I have released v1.0.4-rc.0 which should fix this issue. If possible, would you please try it out with your config and let me know if it works?

@ramses999
Copy link

ramses999 commented Feb 25, 2025

Unfortunately I can't reproduce the problem, now everything works with the old build. Maybe there was network degradation yesterday and k8s responded slowly.

I understand correctly that I should get an error, then set the STEAMPIPE_PLUGIN_START_TIMEOUT variable with increased timing and after that everything should work?

@ramses999
Copy link

Screenshot 2025-03-08 at 12 06 56 @pskrbasu

Hi. Either this doesn't work, or I'm doing something wrong.
The "STEAMPIPE_PLUGIN_START_TIMEOUT" parameter doesn't react at all right now.

@ramses999
Copy link

@pskrbasu Sorry, I didn't take into account that dpkg puts it in directory /usr/bin/steampipe , and I launch the service from directory /usr/local/bin/
It fix it for me sudo ln -s /usr/bin/steampipe /usr/local/bin/

In fact, everything works, I suggest that you can merge it into Main

Screenshot 2025-03-10 at 18 18 07

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants