You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi! I'm trying to build a data pipeline that writes data to Avro files, and when an individual file reaches a certain size, I want to close it and start a new file. Currently I have to check the file size after every write (records are not fixed-size), but if OCFWriter.appendDataIntoBlock returned the number of bytes written through OCFWriter.Append, I could just add that number to an accumulator instead of making repeated calls to os.File.Stat().
Hi! I'm trying to build a data pipeline that writes data to Avro files, and when an individual file reaches a certain size, I want to close it and start a new file. Currently I have to check the file size after every write (records are not fixed-size), but if
OCFWriter.appendDataIntoBlock
returned the number of bytes written throughOCFWriter.Append
, I could just add that number to an accumulator instead of making repeated calls toos.File.Stat()
.The exact line where the number of bytes written is thrown away is https://github.com/linkedin/goavro/blob/master/ocf_writer.go#L227
Thank you!
The text was updated successfully, but these errors were encountered: