How to get the forecasted data without using all the historical data with DeepAr? For applications that involve databases

[Msaleh87]

Hi,

So I trained my model on the training data already and used the following code to compare the forecast with the test data, which is working fine. However, I am trying to extract the forecasted data only without using test data. I tried using fewer historical data, as I don't want to flood the database I am using, but the prediction accuracy became worse:

I have an online application, where I predict the next 48 hours (587 points). I do this prediction every time step (5 min). So I guess I have two questions:

1. can I use fewer data points of the historical data to make a prediction (I tried putting in the test data set the (context length + prediction horizon) length of the historical data tp predict, but the accuracy of the prediction became bad. So do I have to use all the historical data in the test data set to get an accurate forecast? Knowing that this will have a direct impact on my database capacity.

2. How to make forecasts with fewer historical datapoints and without using the test data set?

Forecast vs test data code (working fine)

test_data = ListDataset(
    [
        {"start": df_input.index[T], "target": df_input.Load[T:Threshold + prediction_horizon], 
         'feat_dynamic_real': [time_feat_df_5min_ready.DofW[T:Threshold + prediction_horizon]], 
        'time_feat': [time_feat_df_5min_ready.DayofWeek[T:Threshold + prediction_horizon],
                     time_feat_df_5min_ready.Hour[T:Threshold + prediction_horizon],]}, 
freq = freq)

forecast_it, ts_it = make_evaluation_predictions(
    dataset = test_data,
    predictor = predictor,
    num_samples = 20)

forecasts = list(forecast_it)
tss = list(ts_it)

ts_e = tss[0]
forecast_e = forecasts[0]
f_length = len(forecast_e.mean)
data = {
        'timestamp': ts_e.index[-f_length:].strftime("%Y-%m-%d %H:%M:%S"),
        'Actual': ts_e[-prediction_horizon:][0].values,
        'DeepAr': forecast_e.mean,
    }
DAr_forecast = pd.DataFrame(data)
DAr_forecast = DAr_forecast.set_index('timestamp')

I tried this line of code:

Trying to extract the forecast without testdata (not working)

forecasts_DDAARR = predictor.predict(dataset = df_input[-588:], prediction_length = 576, lead_time = 0,num_samples = 20) 
#list(forecasts_DDAARR)
next(iter(forecasts_DDAARR))

I get an error that the"'str' object has no attribute 'copy" referring to an inner function. I have been trying and I did dir on the function to try different commands, but nothing. I think it should be simple, but I need some guidance here as to what to try.

Thanks

[Msaleh87]

When using the predictor.predict function, you need to update the training data input to it, if you are using feat_dynamic_real.

The reason is the predict function expects the future features inside feat_dynamic_real to be able to predict. Therefore, the code should be as follows:
The key is how to setup the training dataset

training_data = ListDataset([{"start": df_input.index[0],
                              "target": df_input.Load[:Threshold],
                              'feat_dynamic_real': [time_feat_df_5min_ready.DofW[:Threshold]], 
                              'time_feat': [time_feat_df_5min_ready.DayofWeek[:Threshold], # doesn't have time as an index
                                            time_feat_df_5min_ready.Hour[:Threshold],] }],
                           freq = freq)

Then train your model normally. After training the model, use the following code, make sure you use the same training dataset, but add to the 'feat_dynamic_real' the prediction horizon. The prediction function needs this data to predict since 'feat_dynamic_real' was set to true in the estimator.

Updated_data = ListDataset([{"start": df_input.index[0],
                              "target": df_input.Load[:Threshold],
                              'feat_dynamic_real': [time_feat_df_5min_ready.DofW[:Threshold+prediction_horizon]],  #<--------
                              'time_feat': [time_feat_df_5min_ready.DayofWeek[:Threshold], # doesn't have time as an index
                                            time_feat_df_5min_ready.Hour[:Threshold],] }],
                           freq = freq)

forecasts_D = predictor.predict(Updated_data)
for f in forecasts_D:
      Forecast_fun_D = f.mean
Forecast_fun_D

Otherwise you would get mismatch in the dimensions of the matrices input to the predictor.predict.
If you are not using 'feat_dynamic_real' then there is no need to update the input data.

Good luck!