Add environment variable for config file path

[sylvinus]

For users with a non-standard config file path, it would be great to be able to set an environment variable like FLINTROCK_CONFIG_PATH to avoid prepending every command with --config ....

[nchammas]

This is a good idea, but there is already a way to set your default configuration by using flintrock configure. Once you do, you don't need to specify --config.

Is there any reason you would prefer the environment variable over using this command?

[sylvinus]

Ah! I guess I wouldn't want an interactive command to manage my config values in a production environment, where configuration is usually immutable and read-only?

Digging into this I discovered that I could use ~/.config/flintrock/config.yaml (that could be nice to document?), but that won't work with multiple clusters to manage.

[nchammas]

~/.config/flintrock/config.yaml is not hard-coded anywhere. Rather, Flintrock delegates selecting an appropriate config file location to Click, which in turn follows the XDG Base Directory Specification. Depending on your environment, the path may be something else.

Perhaps Flintrock should take a cue from git config and let you set your defaults programmatically.

Another thing we could do is alter the behavior of flintrock configure (perhaps with a new option) to let you load a config from somewhere into the default path.

Could you share more details on your workflow? It would be helpful to understand how you use Flintrock so I can see the bigger picture.

but that won't work with multiple clusters to manage

Could you elaborate on what you mean here?

[sylvinus]

Of course!

I plan to use flintrock exclusively from a Docker container, with a mostly read-only filesystem. That makes using flintrock configure somewhat unpractical.

I'd also want to be able to manage multiple types of clusters at once (different instance types & number of slaves).

So my "ideal" workflow would be:

$ export FLINTROCK_CONFIG_PATH=./configs/cluster1.yaml
$ flintrock launch
$ flintrock login, etc

[nchammas]

Hmm, OK. Here's what I would recommend:

Since you know where Flintrock is going to place the config file (and you control the environment), you can copy your config there directly and get your defaults in place.

I'd also want to be able to manage multiple types of clusters at once (different instance types & number of slaves).

Regarding this point, and your earlier remark about needing multiple config files for multiple clusters, I would say that -- normally -- you wouldn't want to use Flintrock that way.

Your config file should capture the environment you will most commonly work in, but Flintrock allows you to override specific entries at the command-line. This was designed as such with the intention that most users would have a fixed key, region, etc. they wanted to work in, but would occasionally want to change things like the instance type or number of slaves for a specific launch invocation.

So, for example:

flintrock launch my-cluster --ec2-instance-type m3.large

This would load all your configs from disk but override only the instance type with m3.large.

Unless you are working in several completely different environments where everything about them is different, and you don't want to have to pass all those differing configs at the command line, you should only need one config file.

Does that make sense?

[sylvinus]

I agree it makes sense to have some settings in a "global" file.

However let's consider the case where one has 2 clusters with different specs (number of slaves, instance types, region, ..) to be used in parallel. I don't think that's a unheard-of case (for me, it's a high-CPU cluster and high-memory one). I want to keep their configurations somewhere, and the more command-line flags there are, the more painful flintrock is to use.

So my solution for this workflow would be to open 2 shells and set a different config env var in each one. But it might not be the simplest solution, or you can also consider this to be beyond flintrock's scope (though I would find that unfortunate!)

thanks,

[nchammas]

I'm going to let this issue sit for some time and marinate. There are a few libraries and standards for selecting default locations for things like config files, and if we're going to add a new Flintrock-specific environment variable we need to make sure it plays nicely with those standards.

As mentioned earlier, Flintrock already follows the XDG base spec on Linux (via Click), which means that you can set XDG_CONFIG_HOME to different directories and use environment-specific config files that way.

# Linux
export XDG_CONFIG_HOME=/folder/that/contains/  #Flintrock/
flintrock launch ...

Give that a shot. It may be enough for you to get by for now.

[DBCerigo]

I'm also interested in this issue.

My use case: I want to version control my config.yaml, and I would rather not add the --config path/to/config.yaml each time I run commands.

I think my current solution will be to symlink my config into the default location -> this is working fine for now.

An environment variable would also solve my use case.

[Fuzzy-sh]

hello all
I am going to configure the config.yaml file :

### continuously received this error
"expected , but found %r" % token.id, token.start_mark)
yaml.parser.ParserError: while parsing a block mapping
in "/home/centos/.config/flintrock/config.yaml", line 2, column 2
expected , but found ''
in "/home/centos/.config/flintrock/config.yaml", line 11, column 7

I really appreciate it if anyone can help

[Fuzzy-sh]

I have tried yaml editor: http://www.yamllint.com/

it said, ready to go. but once I launch it, receiving error like this:**

[nchammas]

@Fazish - It looks like you're reporting a new issue. Please open a new issue instead of commenting on an existing one.

Also, I recommend you a) delete any existing Flintrock installation you have, b) delete any existing config files you have, c) reinstall Flintrock, and d) recreate your config file using flintrock configure.

If you still have issues, please report them in a new issue.

[Fuzzy-sh]

@nchammas -Thanks a million. I will try to renew them all. in the case facing the same error.
will open a new issue.

[pwsiegel]

I'd also like to "thumbs up" this issue. My situation is that I use flintrock with multiple different AWS accounts in multiple different regions, and this means that the AMI's, VPC's, subnets, security groups, instance profiles, tags, etc. are all different. This has implications on more than just the launch command - if i create a cluster using a non-default config file then i can't use describe, destroy, add-slaves, etc. for that cluster without the --config flag, which gets kind of cumbersome. My solution is to resolve the issue manually by essentially aliasing flintrock to flintrock --config "$FLINTROCK_CONFIG_PATH" - this works fine for me, but maybe it's evidence that there is value in officially supporting an environmental variable or something similar.

(A related problem, which maybe should be a separate issue, is that it would be super helpful to support a profile field in the config file - this would also help users who work in multiple AWS accounts.)

Thank you for building this tool - it has been a huge help!