Skip to content

Commit 74a9b2d

Browse files
frascufrascu
authored
Data upload bigger than 4 MB - Chunked Upload (Azure#19182)
* upload data bigger than 4 MB * fix embedme lines and add example in the README.md * minor fix * make uploadDataToStorageBiggerThan4MB public
1 parent 39a7a66 commit 74a9b2d

File tree

2 files changed

+94
-32
lines changed

2 files changed

+94
-32
lines changed

‎sdk/storage/azure-storage-file-share/README.md‎

Lines changed: 65 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ https://myaccount.file.core.windows.net/myshare/mydirectorypath/myfile
116116
### Handling Exceptions
117117
Uses the `shareServiceClient` generated from [shareSeviceClient](#share-service) section below.
118118

119-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L223-L227 -->
119+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L224-L228 -->
120120
```java
121121
try {
122122
shareServiceClient.createShare("myShare");
@@ -160,7 +160,7 @@ Note that metadata names preserve the case with which they were created, but are
160160
The File Share Service REST API provides operations on accounts and manage file service properties. It allows the operations of listing and deleting shares, getting and setting file service properties.
161161
Once you have the SASToken, you can construct the `shareServiceClient` with `${accountName}`, `${sasToken}`
162162

163-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L53-L55 -->
163+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L54-L56 -->
164164
```java
165165
String shareServiceURL = String.format("https://%s.file.core.windows.net", ACCOUNT_NAME);
166166
ShareServiceClient shareServiceClient = new ShareServiceClientBuilder().endpoint(shareServiceURL)
@@ -171,7 +171,7 @@ ShareServiceClient shareServiceClient = new ShareServiceClientBuilder().endpoint
171171
The share resource includes metadata and properties for that share. It allows the opertions of creating, creating snapshot, deleting shares, getting share properties, setting metadata, getting and setting ACL (Access policy).
172172
Once you have the SASToken, you can construct the file service client with `${accountName}`, `${shareName}`, `${sasToken}`
173173

174-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L60-L62 -->
174+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L61-L63 -->
175175
```java
176176
String shareURL = String.format("https://%s.file.core.windows.net", ACCOUNT_NAME);
177177
ShareClient shareClient = new ShareClientBuilder().endpoint(shareURL)
@@ -182,7 +182,7 @@ ShareClient shareClient = new ShareClientBuilder().endpoint(shareURL)
182182
The directory resource includes the properties for that directory. It allows the operations of creating, listing, deleting directories or subdirectories or files, getting properties, setting metadata, listing and force closing the handles.
183183
Once you have the SASToken, you can construct the file service client with `${accountName}`, `${shareName}`, `${directoryPath}`, `${sasToken}`
184184

185-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L68-L70 -->
185+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L69-L71 -->
186186
```java
187187
String directoryURL = String.format("https://%s.file.core.windows.net", ACCOUNT_NAME);
188188
ShareDirectoryClient directoryClient = new ShareFileClientBuilder().endpoint(directoryURL)
@@ -193,7 +193,7 @@ ShareDirectoryClient directoryClient = new ShareFileClientBuilder().endpoint(dir
193193
The file resource includes the properties for that file. It allows the operations of creating, uploading, copying, downloading, deleting files or range of the files, getting properties, setting metadata, listing and force closing the handles.
194194
Once you have the SASToken, you can construct the file service client with `${accountName}`, `${shareName}`, `${directoryPath}`, `${fileName}`, `${sasToken}`
195195

196-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L77-L79 -->
196+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L78-L80 -->
197197
```java
198198
String fileURL = String.format("https://%s.file.core.windows.net", ACCOUNT_NAME);
199199
ShareFileClient fileClient = new ShareFileClientBuilder().connectionString(CONNECTION_STRING)
@@ -218,6 +218,7 @@ The following sections provide several code snippets covering some of the most c
218218
- [Copy a File](#copy-a-file)
219219
- [Abort copy a File](#abort-copy-a-file)
220220
- [Upload data to Storage File](#upload-data-to-storage)
221+
- [Upload data bigger than 4 MB to Storage File](#upload-data-bigger-than-4-mb-to-storage)
221222
- [Upload file to Storage File](#upload-file-to-storage)
222223
- [Download data from file range](#download-data-from-file-range)
223224
- [Download file from Storage File](#download-file-from-storage)
@@ -235,7 +236,7 @@ The following sections provide several code snippets covering some of the most c
235236
Create a share in the Storage Account. Throws StorageException If the share fails to be created.
236237
Taking a ShareServiceClient in KeyConcept, [`${shareServiceClient}`](#share-services).
237238

238-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L83-L84 -->
239+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L84-L85 -->
239240
```Java
240241
String shareName = "testshare";
241242
shareServiceClient.createShare(shareName);
@@ -244,7 +245,7 @@ shareServiceClient.createShare(shareName);
244245
### Create a snapshot on Share
245246
Taking a ShareServiceClient in KeyConcept, [`${shareServiceClient}`](#share-services).
246247

247-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L88-L90 -->
248+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L89-L91 -->
248249
```Java
249250
String shareName = "testshare";
250251
ShareClient shareClient = shareServiceClient.getShareClient(shareName);
@@ -254,7 +255,7 @@ shareClient.createSnapshot();
254255
### Create a directory
255256
Taking the [`${shareClient}`](#create-a-snapshot-on-share) initialized above, [`${shareClient}`](#share).
256257

257-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L94-L95 -->
258+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L95-L96 -->
258259
```Java
259260
String dirName = "testdir";
260261
shareClient.createDirectory(dirName);
@@ -263,7 +264,7 @@ shareClient.createDirectory(dirName);
263264
### Create a subdirectory
264265
Taking the directoryClient in KeyConcept, [`${directoryClient}`](#directory).
265266

266-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L99-L100 -->
267+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L100-L101 -->
267268
```Java
268269
String subDirName = "testsubdir";
269270
directoryClient.createSubdirectory(subDirName);
@@ -272,7 +273,7 @@ directoryClient.createSubdirectory(subDirName);
272273
### Create a File
273274
Taking the directoryClient in KeyConcept, [`${directoryClient}`](#directory) .
274275

275-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L104-L106 -->
276+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L105-L107 -->
276277
```Java
277278
String fileName = "testfile";
278279
long maxSize = 1024;
@@ -282,39 +283,39 @@ directoryClient.createFile(fileName, maxSize);
282283
### List all Shares
283284
Taking the shareServiceClient in KeyConcept, [`${shareServiceClient}`](#share-services)
284285

285-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L110-L110 -->
286+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L111-L111 -->
286287
```Java
287288
shareServiceClient.listShares();
288289
```
289290

290291
### List all subdirectories and files
291292
Taking the directoryClient in KeyConcept, [`${directoryClient}`](#directory)
292293

293-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L114-L114 -->
294+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L115-L115 -->
294295
```Java
295296
directoryClient.listFilesAndDirectories();
296297
```
297298

298299
### List all ranges on file
299300
Taking the fileClient in KeyConcept, [`${fileClient}`](#file)
300301

301-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L118-L118 -->
302+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L119-L119 -->
302303
```Java
303304
fileClient.listRanges();
304305
```
305306

306307
### Delete a share
307308
Taking the shareClient in KeyConcept, [`${shareClient}`](#share)
308309

309-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L122-L122 -->
310+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L123-L123 -->
310311
```Java
311312
shareClient.delete();
312313
```
313314

314315
### Delete a directory
315316
Taking the shareClient in KeyConcept, [`${shareClient}`](#share) .
316317

317-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L126-L127 -->
318+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L127-L128 -->
318319
```Java
319320
String dirName = "testdir";
320321
shareClient.deleteDirectory(dirName);
@@ -323,7 +324,7 @@ shareClient.deleteDirectory(dirName);
323324
### Delete a subdirectory
324325
Taking the directoryClient in KeyConcept, [`${directoryClient}`](#directory) .
325326

326-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L131-L132 -->
327+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L132-L133 -->
327328
```Java
328329
String subDirName = "testsubdir";
329330
directoryClient.deleteSubdirectory(subDirName);
@@ -332,7 +333,7 @@ directoryClient.deleteSubdirectory(subDirName);
332333
### Delete a file
333334
Taking the directoryClient in KeyConcept, [`${directoryClient}`](#directory) .
334335

335-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L136-L137 -->
336+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L137-L138 -->
336337
```Java
337338
String fileName = "testfile";
338339
directoryClient.deleteFile(fileName);
@@ -341,7 +342,7 @@ directoryClient.deleteFile(fileName);
341342
### Copy a file
342343
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) with string of source URL.
343344

344-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L141-L143 -->
345+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L142-L144 -->
345346
```Java
346347
String sourceURL = "https://myaccount.file.core.windows.net/myshare/myfile";
347348
Duration pollInterval = Duration.ofSeconds(2);
@@ -351,25 +352,57 @@ SyncPoller<ShareFileCopyInfo, Void> poller = fileClient.beginCopy(sourceURL, nul
351352
### Abort copy a file
352353
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) with the copy info response returned above `${copyId}=[copyInfoResponse](#copy-a-file)`.
353354

354-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L147-L147 -->
355+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L148-L148 -->
355356
```Java
356357
fileClient.abortCopy("copyId");
357358
```
358359

359360
### Upload data to storage
360361
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) with data of "default" .
361362

362-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L151-L153 -->
363+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L152-L154 -->
363364
```Java
364365
String uploadText = "default";
365366
InputStream data = new ByteArrayInputStream(uploadText.getBytes(StandardCharsets.UTF_8));
366367
fileClient.upload(data, uploadText.length());
367368
```
368369

370+
### Upload data bigger than 4 MB to storage
371+
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) with data of "default" .
372+
373+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L232-L256 -->
374+
```Java
375+
byte[] data = "Hello, data sample!".getBytes(StandardCharsets.UTF_8);
376+
377+
long chunkSize = ShareFileAsyncClient.FILE_DEFAULT_BLOCK_SIZE;
378+
if (data.length > chunkSize) {
379+
for (int offset = 0; offset < data.length; offset += chunkSize) {
380+
try {
381+
// the last chunk size is smaller than the others
382+
chunkSize = Math.min(data.length - offset, chunkSize);
383+
384+
// select the chunk in the byte array
385+
byte[] subArray = Arrays.copyOfRange(data, offset, (int) (offset + chunkSize));
386+
387+
// upload the chunk
388+
fileClient.uploadWithResponse(new ByteArrayInputStream(subArray), chunkSize, (long) offset, null, Context.NONE);
389+
} catch (RuntimeException e) {
390+
logger.error("Failed to upload the file", e);
391+
if (Boolean.TRUE.equals(fileClient.exists())) {
392+
fileClient.delete();
393+
}
394+
throw e;
395+
}
396+
}
397+
} else {
398+
fileClient.upload(new ByteArrayInputStream(data), data.length);
399+
}
400+
```
401+
369402
### Upload file to storage
370403
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) .
371404

372-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L157-L158 -->
405+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L158-L159 -->
373406
```Java
374407
String filePath = "${myLocalFilePath}";
375408
fileClient.uploadFromFile(filePath);
@@ -378,7 +411,7 @@ fileClient.uploadFromFile(filePath);
378411
### Download data from file range
379412
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) with the range from 1024 to 2048.
380413

381-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L162-L164 -->
414+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L163-L165 -->
382415
```Java
383416
ShareFileRange fileRange = new ShareFileRange(0L, 2048L);
384417
OutputStream stream = new ByteArrayOutputStream();
@@ -388,7 +421,7 @@ fileClient.downloadWithResponse(stream, fileRange, false, null, Context.NONE);
388421
### Download file from storage
389422
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) and download to the file of filePath.
390423

391-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L168-L169 -->
424+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L169-L170 -->
392425
```Java
393426
String filePath = "${myLocalFilePath}";
394427
fileClient.downloadToFile(filePath);
@@ -397,15 +430,15 @@ fileClient.downloadToFile(filePath);
397430
### Get a share service properties
398431
Taking a ShareServiceClient in KeyConcept, [`${shareServiceClient}`](#share-services) .
399432

400-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L173-L173 -->
433+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L174-L174 -->
401434
```Java
402435
shareServiceClient.getProperties();
403436
```
404437

405438
### Set a share service properties
406439
Taking a ShareServiceClient in KeyConcept, [`${shareServiceClient}`](#share-services) .
407440

408-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L177-L182 -->
441+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L178-L183 -->
409442
```Java
410443
ShareServiceProperties properties = shareServiceClient.getProperties();
411444

@@ -418,7 +451,7 @@ shareServiceClient.setProperties(properties);
418451
### Set a share metadata
419452
Taking the shareClient in KeyConcept, [`${shareClient}`](#share) .
420453

421-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L186-L187 -->
454+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L187-L188 -->
422455
```Java
423456
Map<String, String> metadata = Collections.singletonMap("directory", "metadata");
424457
shareClient.setMetadata(metadata);
@@ -427,15 +460,15 @@ shareClient.setMetadata(metadata);
427460
### Get a share access policy
428461
Taking the shareClient in KeyConcept, [`${shareClient}`](#share)
429462

430-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L191-L191 -->
463+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L192-L192 -->
431464
```Java
432465
shareClient.getAccessPolicy();
433466
```
434467

435468
### Set a share access policy
436469
Taking the shareClient in KeyConcept, [`${shareClient}`](#share) .
437470

438-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L195-L199 -->
471+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L196-L200 -->
439472
```java
440473
ShareAccessPolicy accessPolicy = new ShareAccessPolicy().setPermissions("r")
441474
.setStartsOn(OffsetDateTime.now(ZoneOffset.UTC))
@@ -447,15 +480,15 @@ shareClient.setAccessPolicy(Collections.singletonList(permission));
447480
### Get handles on directory file
448481
Taking the directoryClient in KeyConcept, [`${directoryClient}`](#directory)
449482

450-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L203-L203 -->
483+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L204-L204 -->
451484
```Java
452485
PagedIterable<HandleItem> handleItems = directoryClient.listHandles(null, true, Duration.ofSeconds(30), Context.NONE);
453486
```
454487

455488
### Force close handles on handle id
456489
Taking the directoryClient in KeyConcept, [`${directoryClient}`](#directory) and the handle id returned above `${handleId}=[handleItems](#get-handles-on-directory-file)`
457490

458-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L208-L209 -->
491+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L209-L210 -->
459492
```Java
460493
String handleId = handleItems.iterator().next().getHandleId();
461494
directoryClient.forceCloseHandleWithResponse(handleId, Duration.ofSeconds(30), Context.NONE);
@@ -464,7 +497,7 @@ directoryClient.forceCloseHandleWithResponse(handleId, Duration.ofSeconds(30), C
464497
### Set quota on share
465498
Taking the shareClient in KeyConcept, [`${shareClient}`](#share) .
466499

467-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L213-L214 -->
500+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L214-L215 -->
468501
```Java
469502
int quotaOnGB = 1;
470503
shareClient.setPropertiesWithResponse(new ShareSetPropertiesOptions().setQuotaInGb(quotaOnGB), null, Context.NONE);
@@ -473,7 +506,7 @@ shareClient.setPropertiesWithResponse(new ShareSetPropertiesOptions().setQuotaIn
473506
### Set file httpheaders
474507
Taking the fileClient in KeyConcept, [`${fileClient}`](#file) .
475508

476-
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L218-L219 -->
509+
<!-- embedme ./src/samples/java/com/azure/storage/file/share/ReadmeSamples.java#L219-L220 -->
477510
```Java
478511
ShareFileHttpHeaders httpHeaders = new ShareFileHttpHeaders().setContentType("text/plain");
479512
fileClient.setProperties(1024, httpHeaders, null, null);

‎sdk/storage/azure-storage-file-share/src/samples/java/com/azure/storage/file/share/ReadmeSamples.java‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
import java.time.Duration;
1212
import java.time.OffsetDateTime;
1313
import java.time.ZoneOffset;
14+
import java.util.Arrays;
1415
import java.util.Collections;
1516
import java.util.Map;
1617

@@ -226,4 +227,32 @@ public void handleException() {
226227
logger.error("Failed to create a share with error code: " + e.getErrorCode());
227228
}
228229
}
230+
231+
public void uploadDataToStorageBiggerThan4MB() {
232+
byte[] data = "Hello, data sample!".getBytes(StandardCharsets.UTF_8);
233+
234+
long chunkSize = ShareFileAsyncClient.FILE_DEFAULT_BLOCK_SIZE;
235+
if (data.length > chunkSize) {
236+
for (int offset = 0; offset < data.length; offset += chunkSize) {
237+
try {
238+
// the last chunk size is smaller than the others
239+
chunkSize = Math.min(data.length - offset, chunkSize);
240+
241+
// select the chunk in the byte array
242+
byte[] subArray = Arrays.copyOfRange(data, offset, (int) (offset + chunkSize));
243+
244+
// upload the chunk
245+
fileClient.uploadWithResponse(new ByteArrayInputStream(subArray), chunkSize, (long) offset, null, Context.NONE);
246+
} catch (RuntimeException e) {
247+
logger.error("Failed to upload the file", e);
248+
if (Boolean.TRUE.equals(fileClient.exists())) {
249+
fileClient.delete();
250+
}
251+
throw e;
252+
}
253+
}
254+
} else {
255+
fileClient.upload(new ByteArrayInputStream(data), data.length);
256+
}
257+
}
229258
}

0 commit comments

Comments
 (0)