[Xet] Basic shard creation

[coyotte508]

cc @Kakulukian @assafvayner for viz, follow up to #1616

Based on https://github.com/huggingface/xet-core/blob/7e41fb0dd7cfb276222b9668d0b97a984647721e/spec/shard.md

Need to handle:

• split into multiple shards when xorb or file info grows too big
• uploading xorbs & shards (and we need to upload xorbs before shards referencing them)

[coyotte508]

@assafvayner need those two functions from the wasm :)

(also , versions of those two or at least the last one with .update would be nice eventually)

[assafvayner]

what do you mean by .update for those functions?

[coyotte508]

where you can feed it data progressively before calling finalize() to get the hash.

[assafvayner]

let's keep the xorb and range hash computation simple and take just an array of items since those have roughly reasonable limit of ~1K items

the file hash I can see the value but we don't have this feature imlpemented in xet-core yet, and might be a while (it's not simple). For now there's just a compute_file_hash function that takes all the chunks at once but we may be able to update that later

[coyotte508]

Hmm I don't see the difference between range hash & file hash, they both have all the chunk hashes for a file no? (the only diff is that file hash has chunk lengths too)

the file hash I can see the value but we don't have this feature imlpemented in xet-core yet, and might be a while (it's not simple)

yes no problem

[assafvayner]

the range hash is at most 1 xorb's worth of hashes (this is a bit odd to explain, that's why we need to write the whole spec).

let's say a file has the following structure:

xorb A chunks 0-1024 (out of 1024)
xorb B chunks 0-500 (out of 1024)
xorb A chunks 1-44

Then the range hashes for the verification section of the shard containing this file info will need to have:

range_hash(xorb_A.chunks_hashes.slice(0, 1025))
range_hash(xorb_B.chunks_hashes.slice(0, 501))
range_hash(xorb_A.chunks_hashes.slice(1, 45))

notice that all the reasonable parameters to the range_hash function are <= number of chunks in a xorb

[coyotte508]

So there's 1 FileVerificationEntry for each FileDataSequenceEntry?

And it's like this?

FileDataSequenceEntry A
FileDataSequenceEntry B
FileDataSequenceEntry C
FileDataSequenceEntry D
FileVerificationEntry A (for FileDataSequenceEntry A)
FileVerificationEntry B
FileVerificationEntry C
FileVerificationEntry D
FileMetadataExt

?

[assafvayner]

yes; I will need to revise the spec to make this clear