Multipart upload

Multipart upload provides a function that splits the object into several parts for uploading. Each part is a contiguous part of the object data. All parts can be uploaded independently and in any order. After the upload of any parts fails, it can be retransmitted separately without affecting the parts that have been successfully uploaded. After all the parts have been uploaded, FDS assembles these parts, and users can access them just like normal objects. In general, if the object is larger than 100MB, consider using a multipart upload.

Advantages of Multipart upload:

1. High throughput — Parts can be uploaded in parallel.
2. Breakpoint pass - When there is a problem with the network, only the failed part needs to be retransmitted.
3. Upload an object without knowing the size in advance — Objects can be created during uploading. For example, the user accepts data through a stream and uploads it to FDS.

Multipart Upload Process

Multipart upload is a three-phase process: Initialize a multipart upload, then upload all parts, and finally complete the multipart upload. When FDS receives the completion of the multipart upload request, it will assemble the parts according to the part List provided in the request.

1. Initialize the multipart upload

After initializing the multipart upload, FDS will return a unique upload ID to indicate this multipart upload. Subsequent uploading, closing, or terminating a multipart upload requires the upload ID.

2. Upload parts

When uploading a part, in addition to specifying the upload ID, you also need to specify the part number. The part number value ranges from 1 to MAX_NUMBERS, which is used to determine the position of the part in the entire object. If you upload a new part with a used part number, the old part will be overwritten. Each fetch part request FDS will return an ETag; the user needs to record these part number and ETag pairs. When you end a multipart upload, you need to send all part numbers and ETags to the FDS.

3. Complete (Or Terminate) multipart upload

When all the parts are uploaded, you need to complete the operation. FDS assembles all parts into an object by adding the part numbers sequentially, which must be continuous and cannot have duplicates. This operation can take a few seconds when the total object size is large. With different parts uploads of the same object (by upload ID), regardless of which is initialized first, the later completed will overwrite the earlier completed.

If you no longer need an multipart upload, you need to terminate this multipart upload to avoid being charged an extra fee.

Mode

There are two modes of fragment upload: DIRECT and MULTI_BLOB; we do not currently support the user selection upload mode.

Limitations

Billing

In addition to the cost of the request, the storage space occupied by parts will be charged. After the multipart upload are complete, the billing standard for the assembled object are the same as for an normal object. After abort, there is no longer a charge for storage space. So, if the part you are uploading is no longer needed, please terminate the multipart upload to avoid charges.

Complete Efficiency

The Complete phase requires the assembly of all parts to an object, which can be time consuming.

API and SDK

Like other requests, FDS provides the Restful API and SDK, referring to the multipart upload section in Object Operations REST API and Object Operation API Signatures and Samples of the object operations API.

Multipart Upload requests can also be initiated via presigned URL. For details, see Presigned URL.