Rearranging examples between flasc and flasc_cookiecutter_template

[paulf81]

This discussion follows up the conversation happening in Issue #48 , on how we distribute examples between flasc and flasc_cookiecutter_template [fct for short for this discussion] ( https://github.com/NREL/flasc_cookiecutter_template ) . @Bartdoekemeijer and @ejsimley , @misi9170 and I had a proposal we wanted you to consider:

(Note this list doesn't necessarily happen in this order, we wouldn't want to delete examples until new ones are in place)

1. We remove the examples based on generated data from FLASC
2. We add examples based on the smarteole data (Issue [Feature]: Rework examples around open Smarteole database #55) (which is already post-processed so would poorly illustrate filtering methods). The examples in FLASC demonstrate features after all post-processing is complete, and is used to illustrate FLASC usage and also in testing out new ideals locally to FLASC.
3. We post somewhere accessible online a csv or similar of synthetic data, and build up examples in fct that use that example dataset. What I like about this approach are:

• This feels like a good stand-in for the normal first step of "downloading" the raw data
• The data can be named in such a way that the first step in fct will be renaming the columns to flasc standard, again, mimicking practice well
• We can inject into the data a problem for each of the filtering methods to detect

4. FCT I think can have examples that duplicate FLASC after pre-processing, I think that's fine, we just don't want 0 examples in flasc because it's really helpful for documentation, and our own testing.

What do you think @Bartdoekemeijer @misi9170 @ejsimley ?

[misi9170]

This sounds good to me! Just to clarify, I think point 1. should say "generated data from FLORIS"

@Bartdoekemeijer, if you agree, I think you could push forward with copying the examples currently in FLASC into FCT. For now, the data could just be generated locally as it currently is in FLASC, but I do like @paulf81's idea that we eventually remove (or optionally remove) the data generation step and just have users download a csv with the dummy data.

Until we get the smarteole data in place to replace the examples in FLASC, we'd still leave the examples with FLORIS-generated data there too (as Paul mentioned, for documentation and testing in the meantime)

[Bartdoekemeijer]

I'm not sure if you have given FCT a trial run yet, but FCT does already have the FLASC examples in there, and they've been improved quite a bit. It also readily includes the raw data, so no need to manually download that from a remote location. Please do give it a try if you have the time -- I'd love to hear your opinion, and it'll help with aligning future steps.

[misi9170]

Ah, you caught me out, I haven't yet tried running FCT yet, sorry! I'll get to that today, this is good motivation.

[Bartdoekemeijer]

Thanks for starting this discussion! I've been working with the demo dataset that we generated manually and I haven't been super satisfied with it. I've also generally been underinvested in developing unit tests for FLASC. This discussion has given me a new perspective and now here's what I think makes most sense:

The FLASC repository is something users pip install and never actually see the contents of. pandas and openfast follow a similar structure: the repository is dedicated to the code and unit tests, no examples. For simple functionality development, we best use synthetic data and made-up toy problems that can directly be used as unit tests: they're quick to run and are perfect to validate functionality. If I am developing complicated features, it is directly for the purpose of using it in one of the datasets I am looking into. In practice, I would be building that functionality in a Cookiecutter-generated project folder, so these new features are easiest integrated back into the FCT repository. I would say, as much as possible, we transform the current examples based on artificial data into simpler unit tests and put them in the flasc/tests folder.

Then, FCT would be the single entry point for any FLASC-related activities.

• Postprocessing and analysis of raw SCADA data fits best in FCT, like it already is. I would want to use real data for raw data processing in FCT, e.g., the ENGIE dataset that OpenOA uses, or https://zenodo.org/record/7212475 or https://zenodo.org/record/5946808. The current artificial toy dataset has some issues and generally I've been struggling too much trying to mimic real-world problems in it. Real-world data would be most insightful for users, I think.
• Analysis of the Smarteole dataset would also best fit into a FCT-style project folder, since again we require a FLORIS model, energy ratio plots, wake steering uplift calculations, and so on. Hosting the Smarteole examples in FCT instead of FLASC also minimizes the presence of duplicate examples.
• However, we cannot have two different example wind farm analyses in a single Cookiecutter template. Instead, this should be an option for the user, 1) create a template with the Smarteole dataset or 2) create a template with the raw SCADA dataset. We can practically implement this in two ways, where I prefer option 1:
  1. Cookiecutter allows user dialogues when creating a project folder from FCT. We can raise the option to the user: create the examples for the Smarteole dataset, or create the examples for the raw SCADA dataset. If the two templates are similar, both can live in the same template/branch: this is an easy thing to do with little overhead. If there are large differences between the two, this can become a bit messy. A big advantage here is that it forces us to keep FCT as generic as possible.
  2. The Smarteole template can live in a different branch in FCT. Users could easily create their project folder for this branch using cookiecutter https://github.com/NREL/flasc_cookiecutter_template --checkout smarteole. Myself and other parties mainly use FCT for model validation, and not so much for toggle tests just yet, so the default can remain the current template aimed at raw SCADA processing, yield assessment, and wake steering uplift assessment. The risk here is that the two branches diverge too much over time.

Let me know what you think. Also happy to hop on a call at some point if that's easier!

[paulf81]

ok @Bartdoekemeijer , @misi9170 and @ejsimley had a chance to talk some more today and think we agree with you this sounds like a good approach. We were thinking this transformation can be sub-divided into a set of separate efforts so going to submit a number of issues to mark out what we think we need to get going on. Thanks for thinking about all this and for proposing this!!

[Bartdoekemeijer]

Great! Again, happy to hop on a call at some point to help plan this out, if that helps!

[paulf81]

ok sounds good! I'll propose some times, thanks!

[Bartdoekemeijer]

Hi all,

Now we're a couple months in, I've been rethinking some of this, and I'm increasingly moving towards option 4 from @paulf81's initial post. I think this still fits in nicely with the work you've been doing.

1. Like you've been saying, FCT is probably too complicated for most users that only want to look at basic functionality and want to learn FLASC. I like the concept of having the examples live in FLASC, and FCT automatically copies over the FLASC examples when creating a new project folder using cookiecutter. That way, examples live in FLASC, are easy to update/test, and FLASC remains a standalone and complete package.

2. The nice thing about FCT is that it includes generic functions like from windfarm_example.models import load_floris and from windfarm_example.wind_rose import load_windrose. This sets us up for very generic and modular code. With two distinct datasets (artificial and smarteole), we have two different farms, and we'd have to adjust these calls to something like from flasc.examples.models import load_floris_smarteole, load_floris_artificial. This is fine. In the long term, perhaps instead of artificial data we can add noise on top of the smarteole data, and then just exclusively work with the smarteole dataset for everything? Not important for now, but could be a good solution in the long term.

OK, so that means the following would need to happen moving forward:

• Update artificial data examples in FLASC with the newer Jupyter Notebooks in FLASC.
• Create flasc.examples folder and add load_floris functions for artificial and Smarteole dataset in similar fashion as FCT. This is not a must because we can just load FlorisInterface with the .json file but it makes for a nicer and cleaner code, I think, and makes it easier to copy and build the FCT scripts from.
• Update FCT to copy over examples and modify as required upon creating a new project folder.

This is pretty much exactly the track you've been on, and what you have been saying, but only now it's settling, ha. I'll take a crack at this and open a PR. Happy to reiterate together.

[Bartdoekemeijer]

See #94

[paulf81]

this sounds great @Bartdoekemeijer thank you!! We will keep up our efforts on parallel tracks!