Further reduce runtime

[jameshiebert]

A full run of the P2A rule engine (for 10km downscaled climate data) for a region clocks in at about 19 minutes, waaaaay too long.

The profiling shows that (appropriately) most of the compute time is spent pulling data out of NetCDF files and calculating statistics.

But there are several ways to potentially provide some further performance gains:

• Precompute percentiles of ensembles of models
  • The engine uses lots of rules like 25th/75th percentiles, so rather than querying 10+ models and then computing it in real time, this step can be done in advance. Could reduce runtime by 90%.
  • These ensemble statistics could be made available by the climate-explorer-backend.
• Grouping queries to like files together
  • 35% of the total runtime is pulling data out of files. If there are multiple reads to the same file, group these together to take advantage of OS filesystem caching
• Only compute stats that are needed
  • This will require an API change (additional parameter to the stats/multistats calls), but the fact that 25% of the computation is computing stats, and each stats call returns 5 different stats, if we only use one of them, there's a potential for a 20% speedup here.
• Optimize multistats queries for search_for_unique_ids.
  • This is fully 26% percent of the run time and I can't see a good reason why those queries would take so long. Again, this has to happen in the climate-explorer-backend

[corviday]

Precompute percentiles of ensembles of models

I spoke with Trevor about one possible way to precompute the 25th and 75th percentiles.

Currently, the 75th percentile of winter precipitation in a specific area is calculated by the following procedure:

1. Spatial mean: for each of the 12 PCIC models, calculate the mean winter precipitation in the selected area
2. Percentile: from the 12 spatial means, calculate the 75th percentile

I proposed an alternate calculation, which reverses the order of the steps, allowing us to do most of the work in advance and speed up the calculation:

1. Percentile: for each grid square, look at the values for all twelve models, and calculate the 75th percentile; store these values in a new gridded dataset
2. Spatial mean: calculate the mean of the values over the selected area in the percentile dataset

Trevor pointed out that the second calculation will produce more extreme values than the first calculation. For example, suppose for a 2-grid-cell area we have four datasets with the following values:

[5, 9] [7, 3] [6, 10] [5, 7]

The current method would first take the spatial mean for each dataset:

[7], [5], [8], [6]

And then select the 75th percentile:

[7]

The proposed method would first take the 75th percentile for each grid square to make a new dataset:

[6, 9]

And then take the spatial average of the resulting dataset:

[7.5]

Trevor has a suspicion that the values produced by the second method may be scaled by a consistent amount from the values produced by the first method, determined by the probability distribution of the original data. If so, we could scale the precomputed percentile dataset by that amount. However, we'd definitely want to do a lot of analysis before putting this method into practice.

[nikola-rados]

Another suggestion could be to reduce the number of multistats calls by removing the model option. multistats is called once per item in the model list by get_varibles().

results = [
    calculate_result(query_data, query_args['variable'], query_args['time'],
                     query_args['timescale'])
    for query_data in [
        query_backend(sesh, model, query_args) for model in models # HERE
    ]
    if not query_data.count(None)
]

When dealing with historical data this does not matter so much as there is only one model but for percentile variables there are 10+. This piles on a huge amount of API calls that may be unnecessary. If we just take out the model parameter we get the data from all the models (@corviday and I briefly tested this).

This does mean that some changes would need to be made to fetch_data.py:

1. Remove the models API call in get_models()
2. Remove the nested loop for the different models in get_variables()
3. Adjust how calculate_result() handles the output of the API calls or create another method that processes the data before handing it off to calculate_result()
4. Add some way to specify the model or model subset (in case we need to handle more than all-or-nothing cases)

Item 3 is really the only one that requires much work at all. I think this could be a great way to reduce the number of multistats calls, which is by far the worst offender.

[jameshiebert]

@nikola-rados That sounds like a great idea with which to experiment. How about we track this attempt in a separate issue and branch, and then we can run performance tests to track the impact. We think it will help, but let's let the data tell us.

[corviday]

A summary of current experiments, based on discussion at today's meeting:

1. Read netCDF files asynchronously to make file access faster - doesn't seem to matter
2. Skip unnecessary statistics calculations to make the API faster - doesn't seem to matter
3. Use a smaller file ensemble to speed up search_for_unique_ids - seems promising
4. Make fewer calls to the backend by batching calls for different models - doesn't seem to matter
5. Precalculate 25th and 75 percentile values - either impossible or requires a lot of extra work
6. Host files on faster storage - improvement is small, but nonzero

[corviday]

The next round of experiments, based on discussion at today's meeting:

1. Host files on faster storage, part 2: put the files in a memory mount - helps a little, but not enough
2. Test how quickly the rules engine would run if it get all its data from a database by making a dummy database and replacing calls to the PCEX backend with database queries - about 2 seconds
3. Build a flask app around the pre-generated data and run the rules engine against it.

Things discussed at the meeting, but not currently being pursued:

1. Extract each variable for each region into a separate file, so rasterio (which may be the sticking point) is not needed
2. Modify multistats or write a new query to return statistics from multiple timestamps at once, so that separate queries do not need to be made for each season, necessitating multiple file opens and area extractions (while this may help a little, the overlap isn't that much)
3. Check the region definitions and see if the points can be thinned out.

[corviday]

Faster storage helps some, but doesn't represent a solution on its own.

[nikola-rados]

I made a very simple database with one table that has 4 columns (index, period, region, value). I populated it with (p2a_regions) x (time_periods) which gave a small set of 153 rows with random values between 1 and 100. I then wrote a super ugly method that would query the database for a value given some region and period. To simulate percentile variables (these need a value from each model) I looped the query 12 times (chosen because pcic12) and just summed the values at the end. Each loop a new set of region and period values were randomly selected.

The run time clocked in at:

real	0m1.569s
user	0m0.993s
sys	0m0.804s

[corviday]

I have re-profiled the code, incorporating the following improvements:

• use the smaller ensemble, which speeds up search_for_unique_ids()
• datafiles are in mounted RAM, which speeds up file access times

Some relatively easy things to try next stand out:

• with the reduce runtime, calculating the median is now taking a visible chunk of time. I think we don't actually need the median. We already tested a version of the backend that only calculates requested statistics, but it didn't make a difference at the time. Maybe now that we've sped things up a lot, it will.
• It spends over half its time counting the size of the cache (getsize, medium blue block). This issue has been solved already by @jameshiebert - all we have to do is upgrade the ce version in requirements.txt

[corviday]

I have re-profiled the code, with the following improvements:

• use the smaller ensemble, which which speeds up search_for_unique_ids()
• datafiles are in mounted RAM, which speeds up opening files
• uses version 1.2.0 of the climate explorer backend, which has much more efficient cache handling

Possible next steps:

1. Calculating medians now takes up approximately 30% of the runtime, and we don't even need them. The obvious thing to try is to test the version of the backend that calculates only requested statistics. We may be able to drop all statistics except mean; I'm not sure. Will have to double check the code.

2. rasterio calculations are taking up nearly a minute, which doesn't seem right. These should be done only twice (there are two grids involved), and then cached. Possible reasons we're losing time to rasterio include the cache not working correctly (check with diagnostic output on the cache), or excessively complex region definitions (look at regions, or check with simpler ones).