ForkingPickler Error When Reading Arrow File

[AlvinLok]

Describe the bug
TypeError caused by EOFError when loading pickle file through ForkingPickler:

2024-07-10 10:46:29,886 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - TF32 format is only available on devices with compute capability >= 8. Setting tf32 to False.
2024-07-10 10:46:29,893 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - Using SEED: 1360904892
2024-07-10 10:46:29,958 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - Logging dir: output\run-1
2024-07-10 10:46:29,961 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - Loading and filtering 1 datasets for training: ['data.arrow']
2024-07-10 10:46:29,962 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - Mixing probabilities: [1]
2024-07-10 10:46:30,642 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - Initializing model
2024-07-10 10:46:30,642 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - Using pretrained initialization from amazon/chronos-t5-small
The speedups for torchdynamo mostly come wih GPU Ampere or higher and which is not detected here.
max_steps is given, it will override any value given in num_train_epochs
2024-07-10 10:46:45,054 - C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py - INFO - Training
0%| | 0/1000 [00:00<?, ?it/s]Traceback (most recent call last):
File "C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py", line 692, in
app()
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\typer\main.py", line 326, in call
raise e
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\typer\main.py", line 309, in call
return get_command(self)(*args, **kwargs)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\typer\core.py", line 661, in main
return _main(
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\typer\core.py", line 193, in _main
rv = self.invoke(ctx)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\typer\main.py", line 692, in wrapper
return callback(**use_params)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\typer_config\decorators.py", line 92, in wrapped
return cmd(*args, **kwargs)
File "C:\Users\alvin\OneDrive\Coding\Python\chronos\chronos-forecasting\scripts\training\train.py", line 679, in main
trainer.train()
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\transformers\trainer.py", line 1932, in train
return inner_training_loop(
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\transformers\trainer.py", line 2230, in _inner_training_loop
for step, inputs in enumerate(epoch_iterator):
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\accelerate\data_loader.py", line 671, in iter
main_iterator = super().iter()
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\torch\utils\data\dataloader.py", line 439, in iter
return self._get_iterator()
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\torch\utils\data\dataloader.py", line 387, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\site-packages\torch\utils\data\dataloader.py", line 1040, in init
w.start()
File "C:\Users\alvin\anaconda3\envs\chronos\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "", line 2, in pyarrow.lib._RecordBatchFileReader.reduce_cython
TypeError: no default reduce due to non-trivial cinit
0%|

(chronos) C:\Users\alvin\OneDrive\Coding\Python\chronos>Traceback (most recent call last):
File "", line 1, in
File "C:\Users\alvin\anaconda3\envs\chronos\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\alvin\anaconda3\envs\chronos\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

Occurs when attempting to fine tune the model:
python chronos-forecasting/scripts/training/train.py --config chronos-forecasting/scripts/training/configs/chronos-t5-small.yaml --model-id amazon/chronos-t5-small --no-random-init --max-steps 1000 --learning-rate 0.001

Steps taken:

1. Spun up new conda environment
2. pip install "chronos[training] @ git+https://github.com/amazon-science/chronos-forecasting.git"
3. Cloned repo into working directory
4. Converted pandas df into arrow with provided function: convert_to_arrow('data.arrow', df.VALUE, df.REF_DATE)
5. Edited config file to point to arrow file
6. set CUDA_VISIBLE_DEVICES=0
7. python chronos-forecasting/scripts/training/train.py --config chronos-forecasting/scripts/training/configs/chronos-t5-small.yaml --model-id amazon/chronos-t5-small --no-random-init --max-steps 1000 --learning-rate 0.001
8. Encountered RuntimeError: fused=True requires all the params to be floating point Tensors of supported devices: ['cuda', 'xpu', 'privateuseone'], so I did pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
9. Ran training again: python chronos-forecasting/scripts/training/train.py --config chronos-forecasting/scripts/training/configs/chronos-t5-small.yaml --model-id amazon/chronos-t5-small --no-random-init --max-steps 1000 --learning-rate 0.001

Environment description
Operating system:
Python version: 3.10.14
CUDA version: 12.2
PyTorch version: 2.3.1+cu121
HuggingFace transformers version: 4.42.3
HuggingFace accelerate version: 0.32.1

Any help is appreciated, I have tried this with multiple fresh conda environments on different machines

[abdulfatir]

It looks like you're using Windows. We haven't really tested this codebase on windows. Could you try the following?

• Set dataloader_num_workers to 0.
• Use another optimizer instead of adamw_torch_fused (try adamw_torch).

[lostella]

@AlvinLok any update on this?

[AlvinLok]

Yes, I'm on Windows. I've made the changes, and now it's a new error:

Traceback (most recent call last):
File "C:\Users\alvinlok\xxx\03 Code\chronos-forecasting\scripts\training[train.py](http://train.py/)", line 694, in
app()
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\typer[main.py](http://main.py/)", line 326, in call
raise e
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\typer[main.py](http://main.py/)", line 309, in call
return get_command(self)(*args, **kwargs)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\click[core.py](http://core.py/)", line 1157, in call
return [self.main(](http://self.main(/)*args, **kwargs)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\typer[core.py](http://core.py/)", line 661, in main
return _main(
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\typer[core.py](http://core.py/)", line 193, in _main
rv = self.invoke(ctx)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\click[core.py](http://core.py/)", line 1434, in invoke
return [ctx.invoke(self.callback](http://ctx.invoke(self.callback/), **ctx.params)/)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\click[core.py](http://core.py/)", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\typer[main.py](http://main.py/)", line 692, in wrapper
return callback(**use_params)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\typer_config[decorators.py](http://decorators.py/)", line 92, in wrapped
return cmd(*args, **kwargs)
File "C:\Users\alvinlok\xxx\03 Code\chronos-forecasting\scripts\training[train.py](http://train.py/)", line 681, in main
trainer.train()
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\transformers[trainer.py](http://trainer.py/)", line 1932, in train
return inner_training_loop(
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\transformers[trainer.py](http://trainer.py/)", line 2230, in _inner_training_loop
for step, inputs in enumerate(epoch_iterator):
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\accelerate[data_loader.py](http://data_loader.py/)", line 677, in iter
next_batch, next_batch_info = self._fetch_batches(main_iterator)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\accelerate[data_loader.py](http://data_loader.py/)", line 631, in _fetch_batches
batches.append(next(iterator))
File "C:\Users\alvinlok\AppData\Local\anaconda3\envs\chronos-2\lib\site-packages\torch\utils\data[dataloader.py](http://dataloader.py/)", line 631, in next
data = self._next_data()
File "C:\Users\alvinlok\AppData\Local\anaconda3\envs\chronos-2\lib\site-packages\torch\utils\data[dataloader.py](http://dataloader.py/)", line 675, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "C:\Users\alvinlok\AppData\Local\anaconda3\envs\chronos-2\lib\site-packages\torch\utils\data_utils[fetch.py](http://fetch.py/)", line 32, in fetch
data.append(next(self.dataset_iter))
File "C:\Users\alvinlok\xxx\03 Code\chronos-forecasting\scripts\training[train.py](http://train.py/)", line 241, in iter
for element in self.base_dataset:
File "C:\Users\alvinlok\xxx\03 Code\chronos-forecasting\scripts\training[train.py](http://train.py/)", line 491, in iter
yield self.to_hf_format(next(iterators
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\gluonts\transform[_base.py](http://_base.py/)", line 111, in iter
yield from [self.transformation(](http://self.transformation(/)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\gluonts\transform[_base.py](http://_base.py/)", line 186, in call
for data_entry in data_it:
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\gluonts\transform[_base.py](http://_base.py/)", line 186, in call
for data_entry in data_it:
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\gluonts[itertools.py](http://itertools.py/)", line 85, in iter
for el in self.iterable:
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\gluonts\dataset[common.py](http://common.py/)", line 424, in call
data = t(data)
File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\gluonts\dataset[common.py](http://common.py/)", line 345, in call
raise GluonTSDataError(
gluonts.exceptions.GluonTSDataError: Array 'target' has bad shape - expected 1 dimensions, got 0.
0%| | 0/1000 [00:00<?, ?it/s]

[abdulfatir]

For this, convert_to_arrow('data.arrow', df.VALUE, df.REF_DATE), can you share how your dataframe looks like?

[AlvinLok]

This is what my df looks like:

REF_DATE	VALUE

0 2010-01-01 84.7
1 2010-02-01 85.3
2 2010-03-01 85.4
3 2010-04-01 85.8
4 2010-05-01 86.8

convert_to_arrow(
path="arrow_files/p32_df_train.arrow",
time_series=p32_df_train.VALUE,
start_times=p32_df_train.REF_DATE,
)

[lostella]

@AlvinLok could you check if the fix proposed in #156 makes it work for you?

[AlvinLok]

no, adding freeze_support() did not have any effect. I am getting the same error: Array 'target' has bad shape - expected 1 dimensions, got 0.

[abdulfatir]

@lostella this one is unrelated. @AlvinLok you're transforming the data incorrectly. Please check the type signature of the function that you're using to transform. convert_to_arrow expects

...
time_series: Union[List[np.ndarray], np.ndarray],
start_times: Optional[Union[List[np.datetime64], np.ndarray]] = None,
...

The first one is a list of 1-D numpy arrays (i.e., a list of time series). The second one is a list of np.datetime64, i.e., a list of start times, one for each time series in the first list. Since we're only using the start_times, time series are expected to be uniformly-spaced.

[abdulfatir]

@AvisP: can you also check that you're transforming the data correctly?

[AvisP]

@abdulfatir That is highly unlinkely, as I am not using any custom data but generated ones using the provided script and code in example. Are there any likely issues that may happen generating data with python kernel-synth.py --num-series 20 --max-kernels 5 and with the following script? Here are the datafiles that I am using to download and verify

from pathlib import Path
from typing import List, Optional, Union

import numpy as np
from gluonts.dataset.arrow import ArrowWriter


def convert_to_arrow(
    path: Union[str, Path],
    time_series: Union[List[np.ndarray], np.ndarray],
    start_times: Optional[Union[List[np.datetime64], np.ndarray]] = None,
    compression: str = "lz4",
):
    if start_times is None:
        # Set an arbitrary start time
        start_times = [np.datetime64("2000-01-01 00:00", "s")] * len(time_series)

    assert len(time_series) == len(start_times)

    dataset = [
        {"start": start, "target": ts} for ts, start in zip(time_series, start_times)
    ]
    ArrowWriter(compression=compression).write_to_file(
        dataset,
        path=path,
    )


if __name__ == "__main__":
    # Generate 20 random time series of length 1024
    time_series = [np.random.randn(1024) for i in range(20)]

    # Convert to GluonTS arrow format
    convert_to_arrow("./noise-data.arrow", time_series=time_series)

[AlvinLok]

@lostella this one is unrelated. @AlvinLok you're transforming the data incorrectly. Please check the type signature of the function that you're using to transform. convert_to_arrow expects

...
time_series: Union[List[np.ndarray], np.ndarray],
start_times: Optional[Union[List[np.datetime64], np.ndarray]] = None,
...

The first one is a list of 1-D numpy arrays (i.e., a list of time series). The second one is a list of np.datetime64, i.e., a list of start times, one for each time series in the first list. Since we're only using the start_times, time series are expected to be uniformly-spaced.

Alright, well I converted it to a numpy array, and removed the start times argument but received the same error

time_series_data = p32_df_train.VALUE.to_numpy()
path = "arrow_files/p32_df_train.arrow"

convert_to_arrow(
path=path,
time_series=time_series_data
)

Error:

File "C:\Users\alvinlok\AppData\Roaming\Python\Python310\site-packages\gluonts\dataset\common.py", line 345, in call
raise GluonTSDataError(
gluonts.exceptions.GluonTSDataError: Array 'target' has bad shape - expected 1 dimensions, got 0.
0%| | 0/1000 [00:00<?, ?it/s]

[abdulfatir]

@AlvinLok It looks like you're passing a single series to the function. You need to pass a list of time series. If you only have a single series, pass it as [time_series_data].