Large File Management (DLO/SLO)

release/20240425
English

Large File Management (DLO/SLO)

Article Summary

Share feedback

Thanks for sharing your feedback!

Available in Classic and VPC.

This section describes how to upload and download large files. Large files can be uploaded simultaneously by splitting them into several segment objects. The maximum size of Segment that can be stored is 5GB and objects exceeding that size must be uploaded by splitting them into several segment objects.

Large file management methods are divided into DLO (Dynamic Large Object) and SLO (Static Large Object) according to the method of managing segment objects with manifest files. A manifest file is a file that stores segment object location information, and a segment object can be accessed as a single object using the manifest file. DLO's manifest file is empty and has an X-Object-Manifest metadata header, in which is the name (path) that all segment objects have in common. SLO's manifest file contains a list of names of all segment objects in JSON format. For a detailed explanation of the differences between DLO and SLO, see the OpenStack Documentation.

Dynamic Large Object(DLO)

This section describes how to upload/download Dynamic Large Objects.

DLO upload

Here's how to upload an object using the DLO method.

Split large files.
- File splitting is not supported by the API.
- The size of the divided files should be set so that the number of divided files does not exceed 10,000.
Please upload the segmented file as a segment object.
- Segment objects must be placed in the same container.
- The segment objects must be have the same prefix in their names.
- Segment objects must have the same prefix in their names. <Example> 0001->0002->0003
- Uploading segment objects is the same as uploading regular objects.
Please upload the manifest file for DLO.
- A manifest file is a file that stores the location information of the segment object.
- The size of the manifest file is 0.
- It must include the X-Object-Manifest header, and the header's value indicates the location of the segment objects. The '/' value should be set at the end.

Caution

Manifest object has metadata of that object, and actual data is linked to the segment object. Therefore, if you delete either the segment object or the manifest object, the object cannot be recovered.

Upload segment object (three 1-byte files)

Here's an example of how you can upload a segment object.

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/DLO/dlo-object/0001 \
  -H 'X-Auth-Token: $token' \
  --data-binary '1'

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 06:20:35 GMT
Content-Length: 0
Etag: c4ca4238a0b923820dcc509a6f75849b
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx3f192e9948504e689285d-005ba09932
X-Openstack-Request-Id: tx3f192e9948504e689285d-005ba09932
Date: Tue, 18 Sep 2018 06:20:34 GMT

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/DLO/dlo-object/0002 \
  -H 'X-Auth-Token: $token' \
  --data-binary '2'

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 06:23:38 GMT
Content-Length: 0
Etag: c81e728d9d4c2f636f067f89cc14862c
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx22f749319f844f3ebb0dc-005ba099e9
X-Openstack-Request-Id: tx22f749319f844f3ebb0dc-005ba099e9
Date: Tue, 18 Sep 2018 06:23:37 GMT

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/DLO/dlo-object/0003 \
  -H 'X-Auth-Token: $token' \
  --data-binary '3'

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 06:24:04 GMT
Content-Length: 0
Etag: eccbc87e4b5ce2fe28308fd9f2a7baf3
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx016a1cb31a684589b2a1d-005ba09a03
X-Openstack-Request-Id: tx016a1cb31a684589b2a1d-005ba09a03
Date: Tue, 18 Sep 2018 06:24:03 GMT

Upload DLO Manifest File

The following describes how to upload a DLO Manifest file:

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/DLO/dlo \
  -H 'X-Object-Manifest: DLO/dlo-object/' \
  -H 'X-Auth-Token: $token' \
  --data-binary ''

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 06:32:30 GMT
Content-Length: 0
Etag: d41d8cd98f00b204e9800998ecf8427e
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx086882957a56457fb39f9-005ba09bfd
X-Openstack-Request-Id: tx086882957a56457fb39f9-005ba09bfd
Date: Tue, 18 Sep 2018 06:32:29 GMT

Download DLO files

Download the manifest file to download the uploaded DLO file. Here's an example of how you can download the manifest file.

curl -i -X GET \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/DLO/dlo \
  -H 'X-Auth-Token: $token' \
  --data-binary ''

HTTP/1.1 200 OK
Content-Length: 3
Etag: "8f481cede6d2ddc07cb36aa084d9a64d"
Accept-Ranges: bytes
Last-Modified: Tue, 18 Sep 2018 06:32:30 GMT
X-Object-Manifest: DLO/dlo-object/
X-Timestamp: 1537252349.91188
Content-Type: application/x-www-form-urlencoded
X-Trans-Id: txca733836612d46978fa6d-005ba09c2c
X-Openstack-Request-Id: txca733836612d46978fa6d-005ba09c2c
Date: Tue, 18 Sep 2018 06:33:16 GMT

123

Static Large Object(SLO)

This section describes the Static Large Object method.

SLO upload

Here's how to upload an object using the SLO method.

Split large files.
- File splitting is not supported by the API.
- The size of the divided files should be set to a size where the number of divided files does not exceed 10,000.
Please upload the segmented file as a segment object.
- There are no restrictions on the location and name of segment objects.
- Uploading segment objects is the same as uploading regular objects.
Please upload the manifest file for SLO.
- The manifest file is a file that contains the information about the location of a segment object.
- When requesting the upload, the following content must be included in a query parameter.
```
?multipart-manifest=put
```
  Segment object information in JSON format must be included in the request body.

Caution

The manifest object has its object meta data and the actual data is linked to the segment object. Therefore, if you delete either the segment object or the manifest object, the object cannot be recovered.

Upload Segment Object (3 files of 1bytes): An example of uploading a segment object is as follows:

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/SLO/slo-object/0001 \
  -H 'X-Auth-Token: $token' \
  --data-binary '1'

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 06:20:35 GMT
Content-Length: 0
Etag: c4ca4238a0b923820dcc509a6f75849b
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx3f192e9948504e689285d-005ba09932
X-Openstack-Request-Id: tx3f192e9948504e689285d-005ba09932
Date: Tue, 18 Sep 2018 06:20:34 GMT

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/SLO/slo-object/0002 \
  -H 'X-Auth-Token: $token' \
  --data-binary '2'

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 06:23:38 GMT
Content-Length: 0
Etag: c81e728d9d4c2f636f067f89cc14862c
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx22f749319f844f3ebb0dc-005ba099e9
X-Openstack-Request-Id: tx22f749319f844f3ebb0dc-005ba099e9
Date: Tue, 18 Sep 2018 06:23:37 GMT

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/SLO/slo-object/0003 \
  -H 'X-Auth-Token: $token' \
  --data-binary '3'

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 07:05:30 GMT
Content-Length: 0
Etag: eccbc87e4b5ce2fe28308fd9f2a7baf3
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx3ed5e77d00a14631b96f4-005ba0a3b9
X-Openstack-Request-Id: tx3ed5e77d00a14631b96f4-005ba0a3b9
Date: Tue, 18 Sep 2018 07:05:29 GMT

Upload SLO Manifest File

Here's an example of how you can upload the SLO manifest file.

curl -i -X PUT \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/SLO/slo?multipart-manifest=put \
  -H 'X-Auth-Token: $token' \
  --data-binary \
	'[{"path": "/SLO/slo-object/0001"},{"path": "/SLO/slo-object/0002"},{"path": "/SLO/slo-object/0003"}]'

HTTP/1.1 201 Created
Last-Modified: Tue, 18 Sep 2018 07:13:29 GMT
Content-Length: 0
Etag: "8f481cede6d2ddc07cb36aa084d9a64d"
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txa8c0de4034be4bd0b85b4-005ba0a598
X-Openstack-Request-Id: txa8c0de4034be4bd0b85b4-005ba0a598
Date: Tue, 18 Sep 2018 07:13:28 GMT

Download SLO files

Download the manifest file to download the uploaded SLO file. Here's an example of how you can download the manifest file.

curl -i -X GET \
  https://kr.archive.ncloudstorage.com/v1/AUTH_{project_id}/SLO/slo \
  -H 'X-Auth-Token: $token' \

HTTP/1.1 200 OK
Content-Length: 3
Accept-Ranges: bytes
Last-Modified: Tue, 18 Sep 2018 07:13:29 GMT
Etag: "8f481cede6d2ddc07cb36aa084d9a64d"
X-Timestamp: 1537254808.62092
X-Static-Large-object: True
Content-Type: application/x-www-form-urlencoded
X-Trans-Id: txb67bc8f4ade24a80a64a7-005ba0a663
X-Openstack-Request-Id: txb67bc8f4ade24a80a64a7-005ba0a663
Date: Tue, 18 Sep 2018 07:16:51 GMT

123

Was this article helpful?

What's Next

Object Life Cycle Management

Table of contents

Dynamic Large Object(DLO)
Static Large Object(SLO)