Dataset configuration examples

[jlamypoirier]

✨ Description

Fix: #150

Add some examples to demonstrate the dataset mechanism

🔍 Type of change

Select all that apply:

• 🐛 Bug fix (non-breaking change that addresses a specific issue)
• 🚀 New feature (non-breaking change that adds functionality)
• ⚠️ Breaking change (a change that could affect existing functionality)
• 📈 Performance improvement/optimization (improves speed, memory usage, or efficiency)
• 🛠️ Code refactor (non-functional changes that improve code readability, structure, etc.)
• 📦 Dependency bump (updates dependencies, including Dockerfile or package changes)
• 📝 Documentation change (updates documentation, including new content or typo fixes)
• 🔧 Infrastructure/Build change (affects build process, CI/CD, or dependencies)

[oleksost]

Shouldn't it be path/to/dataset_0/fast_llm_config_training.yaml? This way its clear that this should point to yaml files created in the previous prepare step.

[jlamypoirier]

Seems like it would be a bit too verbose?

[oleksost]

Same as before, for clarity it may be better to use path/to/dataset/fast_llm_config_training.yaml?

[oleksost]

Question: In the past we were creating data mixes as a separate yaml file using fml-ops functionality (e.g. /mnt/datasets/dolmafw70_fw30_merged.json). Can a dataset with type file still be pointing to one of such json files?

[jlamypoirier]

No, the json files are deprecated and only available with legacy dataset configs

[oleksost]

Which dataset wrappers are currently available? Here we only document blended, file and sampled. Is the concatenated_memmap (will be deprecated) and memmap still available? Would it make sense to document each of the available wrappers?

[oleksost]

If I have a directory with a bunch of shards (shard_*.bin and shard_*.idx files) and the corresponding fast_llm_dataset.json (<- generated by the prepare command) file in it, is it equivalent to use:

type: file
path: path/to/directory/fast_llm_dataset.json

and

type: memmap
path: path/to/directory/shard

and

type: concatenated_memmap
path: path/to/directory/

[tscholak]

I think what's missing here is an example that shows how to turn a bunch of idx and bin files into a yaml or json dataset config file.

[tscholak]

let's merge this into #146 and then push it over the finish line