Large datasets

[dfdx]

How do we deal with datasets that are too large to be read into an Array? Something of 5-50Gb, for example. Are there any tools for it or earlier discussion?

I thought about:

• Iterators instead of arrays. Pros: simple. Cons: some tools (e.g. from MLDataUtils) may require random access to elements of a dataset.
• New array type with lazy data loading. Maybe memory-mapped array, maybe something more custom. Pros: exposes AbstractArray interface, so existing tools will work. Cons: some tools and algorithms may expect data to be in memory while for disk-based arrays their performance will drop drastically.
• Completely custom interface. PyTorch's datasets/dataloaders may be a good example. Pros: flexible, easy to provide fast access. Cons: most functions from MLDataUtils will break.

[Evizero]

From a design standpoint MLDataUtils, or more specifically the backend MLDataPattern, is equipped for iterators. http://mldatapatternjl.readthedocs.io/en/latest/introduction/design.html#what-about-streaming-data. Its just that we don't have any example implementations for something like that yet
Other than RandomObs etc. that is, but those are more of a "data container decorator" that transform data containers into iterators.

Alternatively it would also be possible to just implement custom data container types for these large datasets that delay data access until it is actually needed. The caveat is that these assumes that at least the indicies fit in memory. The upside is they work seamlessly with partitioning etc. http://mldatapatternjl.readthedocs.io/en/latest/documentation/container.html

My goal with MLDataPattern was to implement the package in such way that it can deal with exactly these use cases cleanly in a "Julian" manner. It can even deal with big labeled datasets where the labels themself would be cheap to store and access (and thus available for resampling) http://mldatapatternjl.readthedocs.io/en/latest/documentation/targets.html#support-for-custom-types

Maybe the MLDataPattern documentation would be a good start for a discussion? I realize its quite verbose and a lot to read, but I tried to be very specific with my definitions there

[dfdx]

Excellent, thanks! I need some time to go trough the docs, so will get back later.

[Evizero]

Maybe @Sheemon7 has some opinions to share on this topic? given that he actually worked with large datatsets in combination with MLDataPattern.

[simonmandlik]

My only experience of large datasets and MLDataPattern was one usecase, when feature array couldn't fit into memory, however simple vector of indices could (dataset comprised around 10^10 examples). I must say that MLDataPattern was smartly designed to cope with such situations.

Are your labels multi-dimensional? If not, I am really curious which type of dataset we're talking about (dataset with so many observations, that even same number of numbers doesn't fit into memory). In that case similar functionality to PyTorch's dataloaders is necessary (isn't supported now, but shouldn't be that hard to implement). In the other case, have you considered similar strategy as with features? You can "index" examples by integers and generate their real "values" on demand. There is a targets method that takes arbitrary function and thus can transform your index of example to its label (similar how getobs works for features). But this is just an idea, if you could provide more specific information, maybe it would help.

[dfdx]

Oh, I didn't mean labels of that size, only feature array. Although I can certainly recall a couple of datasets of dozens of terabyte with labels of several thousand gigabytes, but for them I would use Hadoop or at least some external database because even reading these data from a single disk would be terribly slow.

Here I mean reasonably large data, labeled or unlabeled. I mostly work with text and images. The dataset that brought me here is Food 101 which contains 101 000 images totaling 5Gb. I can work with it as is with my 16Gb RAM, but there might be people who can't. I also have a couple of face datasets that are 10-24Gb which can only be loaded lazily.

(I'm in the middle of reading the docs, so no useful comments yet)

[oxinabox]

Some musings:

The goal of MLDatasets (at its heart) is to get the data into a form that can be processed by MLDataUtils.
Since if MLDataUtils can get at the data, that (proof by construction) shows that a julia program can get at the data.

We can think of data as coming in 3 sizes, these sizes actually depend on quality of hardware commonly in use in ML.

• Small Data: it fits in your RAM (right now this is <8Gb on disk size, corresponding to <32Gb in memory)
• Medium Data: it fits on one computer (so like 8 hard disks for 8TB)
• Big Data: It does not fit on one computer.

From that:

• Small data is easy, and covered by MLDatasets
• Big data is out of scope, it needs a whole bunch of networking magic, loading is a concern for a different package, and even working with it on a single machine may not make sense anyway.
• Medium data is thus the question.

Julia provides in Base (in 0.7 in stdlib) memory mapped arrays.
I've not tried them.
I think though they make a lot of sense as the best way to load data that naturally loads well into arrays.
Though they can only map single files, and the file needs to be formatted as the right kind of binary data.
Any data that is small enough to fit in a single file is probably small enough to fit in RAM, so is small data.

I think there might be scope for a MediumData.jl (or OnDiskDatasets.jl),
which handles the shenanigans of providing an array view on to multi-file datasets that live on disk.
It shouldn't actually be too insane.
Something like

• Base Memory Mapped Arrays,
• plus MappedArrays.jl to handle the data not looking right,
• plus CatViews.jl to bang all the different files of data into one array.

Seems viable, seems like it deserves its own package separate from MLDataUtils,
that MLDataUtils REQUIRES and uses its domain information to say what kind of Mappings need to be used on the data to have it in a good form for use.

Now handling this data with an iterator is easy.
All you need is an Iterator of FileIDs, then use coroutines to express how data comes out of those.
CorpusLoaders.jl is packed full of that.
I think this type of solution is completely in-scope for MLDatasets.

(CorpusLoaders.jl is basically my own shot at MLDatasets for natural language datasets (though there is some overlap), it needs updating at the moment since it was not made with MLDataUtils in mind, and I might want to reconsider some designed decision based on that.)

Since MLDataUtils can process things that are iteration based, or things that are indexed based that means all is good to provide either.
Providing Iteration based means people with more RAM can collect it and then have it indexed based.
Providing Indexed based leads to more feature-ful MLDataUtils experience right now, right?
But as said it is work to do and I don't think there are any current really good solutions to it.
I could be wrong though.

[dfdx]

@oxinabox Thanks for a detailed comment!

Though they can only map single files, and the file needs to be formatted as the right kind of binary data.

I immediately 2 limitations here:

• file system should support large files, so FAT32 is out of scope (yes, I still use it e.g. on USB stick since it's the only one compatible with all systems I meet)
• most likely you aren't going to delete original data (e.g. a bunch of images), so basically you have to double used space on disk - one copy for original images and one for the array (for compressed image formats like JPG array file will be even larger)

What I like about MLDataPatterns is that data containers (which give you access to most of not all the cool features) require you to define only 2 functions - nobs() and getobs(), and both can be efficiently implemented for on-disk data (at least for images). I will try to implement it for my dataset and see how it works.

[oxinabox]

Yes and yes, I agree.

file system should support large files, so FAT32 is out of scope (yes, I still use it e.g. on USB stick since it's the only one compatible with all systems I meet)

Multifile data is a must.
Data is almost always broken up into files <4GB, because all kinds of things break if it isn't.
and it is faster to download in pieces.

One supports multiple files in iteration trivially.
A theoretical OnDiskDataSets.jl package would support multiple file mmapped arrays via something like CatViews.jl

most likely you aren't going to delete original data (e.g. a bunch of images), so basically you have to double used space on disk - one copy for original images and one for the array (for compressed image formats like JPG array file will be even larger)

Yes.
For some formats maybe something lazy along the lines of MappedArrays.jl could be done.
a kind of MappedReducedArrays.jl maybe is sufficient.
Which looks at a field as a mmapped array, applies a map and a reduce function something like a kernal ideally that only needs local information (but this might not be possible for all file types) and exposes its result as a array "view" of different dimensions to the file on disk.
(idk that all reductions that are useful are invertable though, so it might be readonly)

This is basically easy in the iteration case, of course.

[dfdx]

I made a pull request for the above mentioned dataset, but results are different from what I expected. Basically, I created 2 new data types - Food101Data and Food101Targets, and defined methods:

getobs(::Food101Data, ::Integer)::Array{UInt8,3}
getobs(::Food101Data, ::Vector)::Array{UInt8,4}
nobs(::Food101Data)::Int

# and the same for Food101Targets

Then I called batchview(data) and looked at the first batch. I expected the first batch to be what getobs(::Food101Data, ::Vector) returns, i.e. Array{UInt8,4} which I can then pass to my training functions. But instead I get:

DataSubset(::Food101.Food101Data, ::UnitRange{Int64}, ObsDim.Undefined())
 100 observations

Is it intended result or I'm doing something wrong?

[Evizero]

Is it intended result or I'm doing something wrong?

That is on purpose. batchview is still still lazy and returns subsets. You can get the data by calling getobs on this subset.

Here is why: if you want to iterate over all batches you can use eachbatch instead, which can make use of a pre-allocated buffer that it reuses for every batch. For the pre-allocation to work your Food101Data has to implement getobs! though.

[dfdx]

Got it, I'll add getobs! and try eachbatch instead of batchview.

Another question about testing. For smaller datasets we can download all the data to check if everything works fine, but a couple of 5-10Gb datasets probably won't make Travis happy. Any other options?

[Evizero]

oh, that's a good question. I don't know. I'll think about it though. I am very open to ideas

[dfdx]

Basically, code for reading a particular datasets is unlikely to change often, so we can write testing code, run it locally and comment out before merging. This way it will be easy to retest it later, but it won't affect Travis or any users not interested in this specific dataset. The only risk factor I see is when 3rd party libraries (like Images.jl in my case) break and we can't see it because tests aren't running automatically.

[oxinabox]

My plan is (and what is partially implemented in CorpusLoaders ):

• Unit tests / Tests on subdatasets that I create by hand, running with Travis every time
• An environment variable controlling if full downloads and tests run

Longer term I plan to make a CI server separately (either self-hosted, or using another service like CircleCI) to run interation tests with the full dataset. (By setting that enviroment variable)

[dfdx]

Using environment variables sounds like a perfect solution!

Regarding subdatasets and custom CI, I guess it's much more involved and may not work for all the users. Or maybe I just don't have experience in doing such things.