Data Manager

Available in VPC

This guide describes the Data Manager interface. Data Manager allows you to view the list and details of datasets within your Workspace.

View Data Manager list

The list of your datasets includes the following information:

• Dataset Title: The name set when uploading the dataset
• Creation date and time: Initial creation date and time
• Operation: Click [dataset detail] to go to the details view interface.

View Data Manager details

You can view details for the selected dataset. The information is organized into tabs.

Overview

View metadata for the selected data set.

Files and Versions

You can view the file list for each directory in the selected dataset.

Use Data Manager SDKs

Data Manager SDKs support the Python-based Huggingface Dataset Interface.
To upload and download datasets using the SDK:

Install the SDK

You can install the SDK by running the following command.

pip install "ncloud-mlx[data-manager]" # double quotes are required

Prerequisites

To use the SDK, you need a Workspace API key with Dataset permissions and the ML expert Platform endpoint. You need reader permissions for viewing and downloading datasets, writer permissions for creating and uploading datasets and creating tags and branches, and admin permissions for deleting datasets. For instructions on how to issue an API Key, see API Key Management. Enter the issued API Key to complete the prerequisites. You can set the endpoint URL using the MLX_ENDPOINT_URL environment variable.

from mlx.sdk.data import login

login("{ API Key }") # MLXP API Key
login("{ API Key }", "{MLX endpoint}") # Specify the endpoint URL at login instead of using an environment variable

Read datasets

To use datasets in training logic, you must load them as dataset classes. For details, see the Huggingface Python SDK official documentation.

To load a local dataset:

from mlx.sdk.data import load_dataset
ds = load_dataset(
    "{ local data path }" #  local data path e.g. "path/to/folder/*"
)

To load a dataset managed in Data Manager:

from mlx.sdk.data import load_dataset
ds = load_dataset(
    "{ Workspace name }/{ dataset name }" # Dataset location e.g. "workspaceA/datasetA"
)

Upload datasets

You can upload datasets using the same method as the Huggingface Dataset interface. For details, see the Huggingface Python SDK official documentation.

Common upload methods are as follows:

push_to_hub

...
ds.push_to_hub(
    repo_id="{ Workspace name }/{ dataset name }"
)
...

upload_file

from huggingface_hub import create_repo, upload_file

path = "{ Workspace name }/{ dataset name }" # Location of the dataset to upload
create_repo(repo_id=path, repo_type="dataset")
upload_file(
    repo_id=path,
    path_or_fileobj="{ local file path }", # Path to the local file to upload
    path_in_repo="path/to/folder/foo.csv", # Remote file path in the dataset
    repo_type="dataset",
)

upload_folder

from huggingface_hub import create_repo, upload_folder

path = "{ Workspace name }/{ dataset name }" # Location of the dataset to upload
create_repo(repo_id=path, repo_type="dataset")
upload_folder(
    repo_id=path,
    folder_path="{ local directory path }", # Path to the local directory to upload
    path_in_repo="path/to/folder", # Remote directory path in the dataset
    repo_type="dataset",
)

Download datasets

To download a dataset to a local disk:

from huggingface_hub import snapshot_download

path = "{ Workspace name }/{ dataset name }" # Location of the dataset to upload
snapshot_download(
    repo_id=path,
    repo_type="dataset",
    local_dir="path/to/folder", # Path to the directory to download to
    local_dir_use_symlinks="auto" # Whether to use symlinks with cache_dir
)

Create tags and branches

When you create a dataset, a unique commit ID is assigned. You can use this commit ID to read a dataset from a specific revision or record a tag for additional information.

To create a tag:

from huggingface_hub import create_tag

path = "{ Workspace name }/{ dataset name }" 
create_tag(
    repo_id=path,
    repo_type="dataset",
    tag="{ tag name to create}",
    revision="{ revision }",  # baseline version. The default is main
    tag_message="{ tag message }"
)

Because metadata like tag messages is immutable, it cannot be edited, but it can be deleted and recreated. To delete a tag:

from huggingface_hub import delete_tag

path = "{ Workspace name }/{ dataset name }" 
delete_tag(
    repo_id=path,
    repo_type="dataset",
    tag="{ tag name to delete}"
)

To create a branch:

from huggingface_hub import create_branch

path = "{ Workspace name }/{ dataset name }" 
create_branch(
    repo_id=path,
    repo_type="dataset",
    branch="{ branch name to create}",
    revision="{ revision }"
)

Delete dataset

from huggingface_hub import delete_repo

path = "{ Workspace name }/{ dataset name }" # dataset to delete
delete_repo(
    repo_id=path,
    repo_type="dataset"
)