Create an evaluation

Available in VPC.

Evaluations interface

In the Evaluations interface, you can assess whether the service model retrieved the appropriate data to answer a question, and you can also evaluate the quality of the generated answer using the evaluation system.

The evaluation creation process consists of the following steps:

• Create an evaluation
  • 1. Evaluation settings
  • 2. Final confirmation
• View evaluation information
• Delete an evaluation

Create an evaluation

To create a service evaluation:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > AI Services > RAG.
2. Click Evaluations menu.
3. Click [Create Evaluation] button.
4. On the evaluation creation page, enter the evaluation settings.
5. Click [Next] button.
6. Review the entered information in the final confirmation step.
7. Click [Finish] button.

1. Evaluation settings

Enter the following information to create an evaluation:

• Select evaluation target Select the service to evaluate.
  • Service: Choose from the list of registered services
  • Service description: Description set when creating the service
• Basic evaluation settings: Enter basic information for the evaluation.
  • Evaluation name: 3–20 characters; duplicate names allowed
  • Evaluation description: Up to 50 characters
• Select evaluation model: Select the model to use for evaluation.
  • Only the default configured model is currently available.
• Evaluation type: Select the Q&A type to evaluate.
  • Only Question and Answer type is currently provided.
• Evaluation metric & dataset Choose how to configure the evaluation dataset.
  • Evaluation metric: Only the default setting metric is available
  • Dataset configuration: Choose how to prepare the evaluation dataset
    • Automatic: System automatically generates the evaluation dataset (additional fees apply)
    • Manual: You manually create and upload the evaluation dataset

2. Final confirmation

Review the information entered in the evaluation settings and click [Finish] button.

View evaluation information

You can review created evaluations in the NAVER Cloud Platform console under AI Services > RAG > Evaluations menu.

To view evaluation details:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > AI Services > RAG.
2. Click Evaluations menu.
3. In the evaluations list, click button for the evaluation you want to view.
4. When the evaluation details section expands, review the evaluation information.
   • Evaluation name: Name entered during creation
   • Evaluation description: Description entered during creation
   • Target service name: Name of the target service selected when creating the evaluation
   • Evaluation target service description: Description of the target service selected when creating the evaluation
   • Evaluation creation date: The date and time the evaluation was created.
   • Evaluation completion date: The date and time the evaluation was completed.
   • Progress status: Indicates the current progress of the evaluation.
     • Generating evaluation set: Displayed when automatic evaluation set generation is selected.
     • Evaluation set generated: Displayed when automatic evaluation set generation has been completed.
     • Evaluation in progress: Displayed after an evaluation set is uploaded.
     • Evaluation completed: The evaluation has finished running.
     • Stopped: The evaluation has been manually stopped.
     • Failed: The evaluation has failed.
     • Dataset generation stopped: Displayed when the stop button is clicked during automatic evaluation set generation.
     • Dataset generation failed: Displayed when an error occurs during automatic evaluation set generation.
     • Waiting for upload: Displayed when manual evaluation set creation is selected.
   • Evaluation set type: The type selected (automatic/manual) in the dataset configuration step during evaluation creation.
   • Automatically generated evaluation set: Displayed when Automatic is selected in the dataset configuration step. (Not displayed when Manual is selected.)

Delete an evaluation

To delete an evaluation from the evaluation list:

1. In the VPC environment of the NAVER Cloud Platform console, navigate to > Services > AI Services > RAG.
2. Click Evaluations menu.
3. In the evaluations list, click button for the evaluation you want to delete.
4. Click menu, then the Delete menu item.
5. In the Delete evaluation popup, click [OK] button.
6. The evaluation is deleted and removed from the list.