Documentation Index

Fetch the complete documentation index at: https://guide.ncloud-docs.com/llms.txt

Use this file to discover all available pages before exploring further.

Serving

Prev Next

VPC 환경에서 이용 가능합니다.

ML expert Platform의 Serving 기능은 Kubeflow KServe 0.17.0 버전을 기반으로 구현되어 있습니다. 자세한 내용은 KServe 공식 문서를 참조해 주십시오.

Predictor 배포

KServe Predictor 배포

참고
  • KServe의 기본 Timeout은 300초로 설정되어 있습니다. 필요시 InferenceService spec에서 변경 가능합니다.
  • Load Balancer의 idle Timeout은 300초로 설정되어 있습니다.
  • 처리 가능한 요청량보다 많은 요청들에 대해서 Queuing이 필요할 경우 target-burst-capacity와 container-concurrency 값을 설정하시기 바랍니다.

Using HTTS

아래는 Google의 공식 tensorflow-serving 모델 서버를 사용하여 Tensorflow 모델 서버를 배포하는 방법입니다.
Kubeflow KServe가 지원하는 다른 모델 포맷을 YAML에 명시하고 사용하는 것도 가능합니다. 자세한 내용은 Predictor Runtimes 정보를 참고하시기 바랍니다.

아래 YAML 파일을 적용하면 InferenceService가 생성됩니다.

# tensorflow.yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: flower-sample
  namespace: p-{projectName}
spec:
  predictor:
    model:
      modelFormat:
        name: tensorflow
      storageUri: "gs://kfserving-examples/models/tensorflow/flowers" #Mount Path 정보 입력
참고
  • InferenceService 이름은 최대 26자까지 설정 가능합니다.
  • Object Storage, Ncloud Storage를 사용할 경우 stroageUri를 S3 Rest API 형식으로 입력해주시기 바랍니다. storageUri 관련 자세한 내용은 Storage Options for Model Artifacts in KServe를 참고하시기 바랍니다.
  • PVC를 사용할 경우 storageUri에 생성한 PVC 이름을 입력해주시기 바랍니다. PVC 관련 자세한 내용은 Deploy InferenceService with a saved model on PVC를 참고하시기 바랍니다.
kubectl apply -f tensorflow.yaml

다음과 같이 InferenceService 목록을 조회할 수 있습니다. 모델 서버가 준비되면 인퍼런스 요청이 가능한 엔드포인트 URL을 확인할 수 있습니다.

$ kubectl get isvc -n <namespace>

NAME                     URL                                                                 READY   PREV   LATEST   PREVROLLEDOUTREVISION                   LATESTREADYREVISION                              AGE
flower-sample            http://flower-sample-predictor.p-{projectName}.svc.cluster.local    True    80     20       flower-sample-predictor-default-6p5cc   flower-sample-predictor-default-7k6jh            10d

해당 URL은 ML expert Platform 내에서만 유효하며, 외부에서는 접근이 불가능합니다. 외부 엔드포인트를 통해 모델 서버에 접근하기 위해서는 IngressRoute 생성이 필요합니다.
아래는 단일 InferenceService로 모든 트래픽을 전달하는 IngressRoute 생성 예시입니다.

참고

IngressRoute에 대한 자세한 설명은 IngressRoute 사용하기를 참고하시기 바랍니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: flower-sample
  namespace: p-{projectName}
spec:
  httpRoutes:
  - route:
    - inferenceServiceName: flower-sample
      weight: 100

IngressRoute가 Provisioned Phase가 되면 다음과 같이 IngressRoute의 엔드포인트를 확인할 수 있습니다.

$ kubectl describe ingressroute flower-sample -n <namespace>

Name:         flower-sample
Namespace:    p-{projectName}
API Version:  endpoint.mlx.navercorp.com/v1alpha1
Kind:         IngressRoute
Spec:
  ...
Status:
  Conditions:
    Last Transition Time:  2025-02-18T14:08:43Z
    Status:                True
    Type:                  IngressRouteReady
  Http Endpoints:
    {projectName}--flower-sample.srv.kpb3r.mlxp.ncloud.com
  Phase:  Provisioned

테스트를 위해 JSON을 input.json 파일로 생성합니다.

{
  "instances": [
    {
      "image_bytes": {
        "b64": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAErASsDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwC9A42ir9vA0nOOKxYJhkDqe1bNvO0ZAYdaIsC8LLjOwH60yWDAwY1/75qzDcDAz0qfhl55BqxGE1pCzZwVPt0qJ7MgZQbh7da1Z7bncBVQgoaVhlGFvKlBIwOhqxPFxkdKmdY5xiQYP94daaqtGPKkO5P4X/pU2AoKMMQatWv+tAPXpTJ4ipyBTVYqwYHBFTezA1ivHNRsuRU1tOlymOBIOo9aVoyGNaCIEHanEEEMKXbg07BAx2NICXO5KrvwcVPEcDFRyD5qTYDYhuPuKnA4waitxmQj1FWGX9Ka2ArODzUXU5qxIM81AODzUtjGzHMfvVRcl6mmOMio4V3PSAtwjBUd60l+6DVCMAzH2q6D8v0qo7CIJ3xmsqQ8kmtC5YAVmyctntSbGRkDOT0qWMFyABUWNzD0q5EuxM9zQgJQAqgCkJxS9vemMasA3c8CpFPHNRBgBkinBvSpuBMGxRnPWo1561IOlMBQMEU2R8DFKW2rk1XdsmgCN+TmqskuHIqeUhVNZMkoZyckZqQILTi5UntzWtHMOVbpWQh2zCr6jIBpRGzUjl2jBPHY1chuSODyKx4pOzdKnVyh68VYjbDBlyvSq88G4bhVeG4Kkc8HrV3eGUEVQjLkUr+FRmQgYzV+aMODxzWdIpU0mMerh1wahdCpPvTN21gQamB3jB+qn1rOQDI5GjcMvBFbdvMt1FkfeHWsJhzU1pcG3nDZ4PWlGVgNd4+MigL8uKscMgdeVNRsAORVsRGFwc1G45qfKg/MM/U0jLG3RQPxNS2BCh2OG9DVxwM57GqxRQc8j9asp80I5zjiiIyu64zVdhxVtwMVVak2BUlOTUlumATTXXmpPux0r6AS2vLv7GrLNtFVbM/K596knbgGqT0AqXLZeqbgsRU8x96hJzgCk2A+JPmA61PA4mUSL9wk7fcetULtmEMdvGSJrltgI6hQMsfwH6kVqRIsUaqgAVQAAOwFUgEJ7UwDOc1Ky55/OmtgcCi4EZ6UqqSc0Hk4p46igB44pQaaM5NI7hVx3qkA2V8nHaoAdzE9hTZHOMd6ZczfZoQq/fNDArahcgAxLyf4iKzs0OxJ5696ZUDQP97NaVsdyg+1IPszHlFzU8SRg4jGB6VSQh3linp02mpQm5enNJs9aoBoynfirMFwVOD0qADjDUn3W9qANIsGGQeKqXCK3PekjlIOCeKfJyN1AGXIMZFNik6xscc5U+hqxMgbPrVFwVas2BezvXOMOPvCo2GD7UyOXOG/iHX3p8hGzdn6Vm0M0rG8ZLYxtzz8pp0lyx/iNZUMpzzVkturURKZGP8AEfzpRMw6Nmq5HvTMspz1pAaUVzzhjiptxjPmRnjuKyBNzzxU8NwUbDcqaXoBreYJU3L+VVn5zTEcRvkHKNUjcE4qZdwITyabK3yGpG4GaqzN+7qG9ALNicwn/eNE75UgU2zb/RQfc0krY4rS+gFZgcc0iKM+9Kc81FcI727QxnDyjbu9AepoWrAZpv8Apt7NqB5jA8mAeig/MfxOPyFa4HFQ20KW8KQxrhEUKB7VYXFWAvlkhSDx1PvUchwSAKlD7Uyep6CoS3UnrU9QGHg0DJ5xSb8mjdjvVAOZ9oqs75JOaJX3Hg1GBmmAKRuLt0Xms24lMshbt2qxezgDyEPuxqkxpNjImo4pGOOarmbk0gJvMINWIbp42BB6VBLC0Z9qjVsGjYDqrWVJ4w6n6j0qcxbh71ztndtbyBlPHcetdLayx3Me+Ns+o7irTuJkDRDvwaYVIODWg0IYc9agkgBGDTEUyCv0pwc4wac8ZTg9KjYFRSAil6ZxVOYZFXGPBBqpKKljIFJB61KzFlqJhTkbPBrO+oxysR0qwrkjk1Wxg1IoPBJ4PpSc7BYkOfU0zzHXvke9WNuFBHQ1A/BrKVRoaQm/dweDShyOOtRZB+tAPHNSq6HYv203/LNuh6VeVvkweq1ioxB+lacUm5Nw7jBrVTUoktEsp+SqNw2F4q1I3yCqM5yQKwchpF62Yi0/GkcHgmkh4gAoOSK1UxEfVuKdbKZH8zseF+nrTGO0qo6scVciXgYrWG1xDxwcUm/GQKc3FVS3J5qmwHmUkknoKYXJGaZuBzTd1JMB4PBzxUUkhPApWcnpTFUk1SAdGrOcVW1S/TTbbIwZW4Rff1rQXbEmSefSqC6bHPdNd3Q86U/dDfdQegFUBkWYurnlI2YseWPetSPSZW5llC+yjJrUVABjt6CnHihJICkulWqcsGf/AHjUwt4QMCGMD/cFSM4HWm+YKq4HO/aAww3NRuqtypxTNpFIMisrjFDFDg1ctbySCQSRsQf5/Wqm4MOaT6GlewHY2OrQXWFciOT0J4NaDRq68/nXBLIRwa0bTVLi2wBIWX0bmqU+4rHSSQcFW5HY1TkiKZBGRUtrq8NyNr8N6VaYJIvysCPar0YjGdMDOeKrSL6ng1pXFuUJxyDVCSMgH0qGMqMvao84NTOp61XYkHmspDRYXBxnpSFjG2DyKbEdwK96eR5ilT95elZSKRYglBHlseD0NJKpU4NUlk2nB6VcjlEq7HPzDofWsJTvox2KzcZojbccGnSrgkVCpIkFcdaorblJFg8VctHzlapBs1Ztf9bTw+K01CUS07dvaqMh3TqKulSSTjPHWqCgGdnzkKOtYLGxbDkZoxnEYqR4j5e4HPqKrwncAO2M1X1jVRp+lXFxn7inHu3QCu6E3JKzIaC1lF3qk+05jtgIh/vnlj+AwPzrXUYArnvB0LLoUMshJknZpXJ7kmuhLAZxXcpIgimYjPNVd3HWpJnLHFQgcYp81xjht7U4AGkGFpDPt4QZNaIQ9YiRnoPemng4Tk+tIFdzmRjj0p5dEHFWmAqRfxOeakyAOwquZ93Sk5PJouBM0uKjMrHpUeRn5qQydhRcB3Pc0vHrUYJanbT6UwMX6ikKelafA/gX8qRljPWJfwzScAMsxjNJtx9K0Ht42HykqffkVA8DpzjI9RzUOLQ7lXbijDdRU2zIppX3qRipJ0z1rQttQkj4Ylh655FZjLzQMryDRewHUwXyTphiDnof8aZcQYO5eRXPw3BQ5U4Na1tf5Xa3I9KtSuTYgkjK5x0qpIpwM9K2HRJlLIefSqEsRGR+lRJAUVcxuCO1WpDlROn4iqknTp0p1tPtYo3KNXJOdtGWlcllUMBIvRuv1oGVOD1FOWIiRIz91zgH0qS6VoNwaMgKeJQMgfWvExGK960TeMRpPmqMnD4796JIQsf7s5lGAQT61YeSOS8SFwFkH3WHRuKq6fFJdajMZ3aMRtgqByT269q4nUnJXb2KskMjUiIS7sl32hfTtWhaArIQ42kdaaY/7PvzG6+dFMSWfHC91+nvS2e+Sab7QORNsUZwG7jn0xmpc3a6FYnDpC4lDEl/4G6DtVa5UqRDEADKTj2Hqas3Nkz6hGZGxCFJLL/FzwKluIykkQgQtC53N3Kn0+maxUrNMvRoiEZWPapAJ71keItPkvdPYiNnjg+cQjOZT/8AW61uOY1cGUcryFzUgkZ0JVsDtXfDGWmm9kZuGhX0qJLXT4IACuyMfKe1WZJQFOKiKlSCWycfnUUjjua9ali41PhZk42EbLHNISFFM83nA5pVXPLV3Ql2IaDBf2FOykQycUySUIMDrVZi0h5NbJiJnuGc4WkVWP3qYo29OtSKrt34q0wHhgo4GaCx/CnLCw561OkJP8NWIrBd/apVtj3qysI74FSZjTimBElsO1S+QKQzf3cCk3v60AUiqEcVG0R7DNPIZT92kLY5yRWgFd0I7UwMyGrLHPYGoXQHkVLAYYUlGUwj/oarMhRyrqQam3FDU6yRzoEl7dG7ioaTAzyuR0phQ+tXJreSLn7yHow6GoNhas2h3K5UA5zSrIUI5qQx5qIoBwRUMZehvMEZbB7GrS3Mdy/lMQk3YH+L6HvWJnHHpSs0c0ZinBKeoOCp9Qe1Tz2CxcvY2jY8VXgYMrhhhf73pTY728geO3ukW9tXztuPusvfDe9aECQmMqE3I/OD1FeVmFaKVlua049yNLooVjLD5MMcjORmrZN3LfmHzVjg2bn3LkMvp9TVeS6htbNXSNTk5RiPSrdhdx3ds0lztIcHJHHHt714cr/FY2JTbwGzDJ80kBym48/5xTZHzLFLLEygryw43+lFvJa/YZF52AZEh5IIpkN6k2yCcOcHdtIxzUaktl10gup02zMqlcFRyM0omRElhk25QYGBzu9qqLav9rl+ySII+q7mzz6UxHiXzYbkL9oHzGQHnd2x7VLj0Fc0Fml+y5kjbY2OT/CakaRSEMY/eE7do71krqBkgWNllCzMOdp/P0q35aQPEYHd9wKlSMnjvUuFtykyeW1ju51WQsjIm38e1V5LWRSiGcDy+Hx/ETzmpxK0kgl2ERYwzHru+lJe2pn8trc4kzyC3BHrSTa0ZRG7IkQIbcvTJ65qsUeZ2H3VXqT6065kkSfyIrZiQMFmHHHU06TKWyq7ZbGSfU16GB91uUnotTOYxIVU/eyaSRivFAxgHOaGw3yn8K9+hVhWjeBi01uQFWY05YSe9L5wXjbUiTqeq4+ldcSWSRwkdRmp1RVGW4qITf3TxSg7jljWqJJRJGvTrSGZj0OBTfK3Hini3brV2Ab5jNxTgCetSCE9+DR5ZxinYBAqjmjzVpPKOOtHlU7AVRLwOhoyHFZyuw6VMsz9aq4FghajYYHBpPNzzRvDD0pMCB8HtioiQOhqV1Peq0kZBz2qWBbgu2i44ZT1U9DVwWttdjdAxjfvH/hWGWZDx0qWO4dCGBII7ip5h2L82nlTgvtP+0KrPZSjkbH+jVft9VhnTyrtQQeN4qO7025C+bp06TL/AM8pDj8mH9RSaT2AyJo2X7yEH3FVJJFUdeadN4hlsJvK1Kxnt+cbiNy/nTzf6XerkBTnupK1yVdFcpDdME8935SofKP3s9vetcmCxTZcDcm4/Oh5APrVWK4kuA0dtHjbznODTzAbiaGV0EinO5Dkfn+NfN4ibqTvLRHTHRE8losVspkKT23VY+4z3z9Ka8FvayRyQiT7G/OGHCmp44omsnW5LbsHdGrcLjgfXtQdQVdOkhYeblNqoozx9K57vYTIbmWAT2zoGUOSGUDCk9jS3Fwl7fpsuFjZUAdgM5PSobm/hmsIbZnVmaQEJjgge/arN1FHdxWoh8qBgx3N/s/TuelVta5DZWN79gmliMgaVMkcEBqnhube4s5FmTMjjcGbruPpisvVIn0u5jE9wkz3PAcLjaKkb7Pp8kEkL7lkByHP3SMcj3q+RWTW7Iua0OpLJYzQyZ+VMFAMMD2xn3plnfva3Dw3aeXMg4UntjrVC9vXk8m+ijXbG20kA7m96fNqCSz29y6EmL5Hcp0zjGah0tNtylI14dTVpZHPIz90jk1Na5aLzvNBc/MExwB6fWq7zW63cExVBI8fGO/PFSXksUcgMZbMv+s2jhPeudxWyNUyO7vfOuo0BYKAGc+vtUVwr3upCFNywxrukfH8q02mha0kAjUqqZB6Z46VAl15kGyFQCVOBn+tVSqcmtgavoQtDDgorlSOnOahkzCu6QjaOd3bFPQmGVEeAmM43yZ6+uKknaPa8IAdG4OfSvewlanJ6JL0MZJmb9ttWOVcyeyKT/TFC3Fy5/0fTZCv964kWIfhjcf0qyMRjCKFHsMU1mYmvWi0ZsehuMfvPIiOekbGT9SF/lUhl2+hqDDkU3Yx5NbJiLH2kjocU8XLf3jVUITRsNXcRdE7H+Ol85/7xqmEYdDThuHencC557g/epftL+oqllqTLU7gQKvNPCkHBNOVkIG5amVI2Aw2KYEITqKTYQfSrnkHqMGlMB/u8UgKTKfc00oSORV4REdRQbfnikwMxoh0IqJodp+WtY24I55NNNvj+GpaAx2hYcg0+C5ntmzHIw9uoNXmtjk4FQSQbe1ZO62KRZTVbe7Qw30KAHuRlTUN5YlSj2RhEWeU8sFT9DVCZAB2qvDcSQMfLkIH90nINcmJqtQa6lRjqaUTNNI1rtWJ8E7umfUVIoffHapcA7Ry5GCB/WoLC4juhKZGjSUHaoc4/HPTFVPImsbo3VyywhMjy2OSwPpivm3FuTT3OnoXLuAxTC3FzuWYFtxGDwelN06eHTmfjYe4cckVMyiOSO4nlWSXd8qfwjIqS6uYZb+1V9nmxhjz9OKm+ljNmJetBJqdw8mYlAHl4UqPekt5tQnuVeOESW6ggEN6VY8Tajb6nHBYxnE5bJbHQVDpMdxbXCWECmYjJGDwfqa6Uv3d2tfP8yGJBqcWpTvHdw7Y0G1S33vfH5VB/ZN5GrtegPDJ8lsQ3IPbP1qxc6PEbGW4Fw0FxHlnjK8euKqy67P/AGfBEkb/ALyQYJGFBHaqjr/D2/IhmlYvcaVcpHeRqpC5A/hI+vrVy1vree/vgPnWQj91jqMAdPrVKG6XVZYoZ2ZPJ+YHg7varl8sVlqdvNZkuZ1KMMZIA57VzySbs9xok0/ZCstvdQsuCRtk6qvbBrQWxKwFldjC7fOCfnVewqnA1trLkytiZMojdCh9SO/NT28k8pksN+LhfldsHGPUf0rCd73+81iaBitksnOPlRSWUnhlxVKxaJnCR2hjTqrO2SfpT2tJlb7MzLJCQCW3YJGf8aa8aWDhZWdl6RcYI+prJbWuakrNiCZdwfa3y4+g/wDr1EkYI5/OkjCG1OxtoLDrT9siDBU49ua9fApaNGM77DvIHrxSiFD1wKZ5rZ9KUEsc170DFj/JGeBSi3p6c8ZqYLnoea3RJWNsMcUwwe1XQhPUU7y+MVYGeYcUhirQ8oDjrSGIdSKYGb5dJ5RrQMIPQU3yPaqAykjPpU6R1l29zOhAT5h/d61s28jSLl49n1NNO4CqpHQ08zGPgcmplVT3psiLTsIbHdIxw6496tCNSuQAR6g1mSpg8Ypsd3JC3ysfpSGanlD0pjBV6kVB9viuE2Sh4mP/AC0jP9Kgk02WXm3vkm/2WO1v8Kl+QE8kkXr+VVnngHVAfqaqTaZqEWS8Dkf7Jz/KqLiQZDRupHqprCU5LoNIvy3cfRUT8qpl43kG9FC9ziqbS7eM0eflSvXNefi5TlHQ1glcnu1DeXFbhDvPDHt70+5toZWG26jeeDBUyc+Zj2HP6UkMUJieGEM08gyAR04/SqcNtDYXsVzPdr52SPKAzgnjqK8OPrt+JsTBvPuRDdRPEoG5se/Sori2jsZZFeYusg3K5649KtXLPeX0QikChMh3xnI9qy9ctZY9RiWdzJbsu5SOBmrpq7tsZyNEHTbfRZF8tRJtIEnVmPaq1ldtpzjdJkS4IYDkN6fSqul6XaXBkMrtuJIQZ4X3qndLKLuWISGUQpuDIMFe3Pr2rRQjJuN7kGhrerx3LyNEMoRtlYfxN7Cqtiw1eeGzm/dQRfOXHXPQAVGLL9y7yFEeNcqAfvZ9ahluvs9ukcKmK5XBwBwRnk1pGKS5Ybg0dJZG0jtpIZo8BWPzg859aS2ubi2vY5bmMIJlzC56FQeh9D7VmWIE9rJO75kzkN0GRWhLNeajosyx2jTBfmIf5SPcVzyjrZiHyXTx628kUQ8iYgGQdA2Oa2pIUWKO5tpHNxwjlv4/rWNp0M2raaFtisYUhgX7MK0tPiluoj9omEc8ZZQo6bgec1hVVvloaRZZmWW1jFzNIhiYbWIJ+Q1E01rqN0oeSR1UcBBhc/jz+lWbWKa8jIZAsQOJFc8v7AVS02IxFHkCozE+WhOSR3rFWs31Rqi7A0LRSLblWBGACw+U++Kqi4uI5WRAzhTjIU4NTzzWUEx/erHu6j3Hf9azl1GVh8x4J4NdeCS573aImaS3Of8AWwMPcCp0MMnKPj2NZi3b/wB6p470j70aNX0tKatqc7RpCNlORyPapApzmq0N1bN/ejPvyKuo6EcSK1dSVyRytinja3UUmFz1FL8o6EVdgHgKKXYDzTMgdx+dAkIosA4xDsKPJpPN9qPNHrTA5tBs4Xge1So5HrTTG5HCmmiKQ1QF2OUHgnFSk8etU44znk4NW1QbeTzTEV5Bmqrrz3rSdFxxUDL3xUtDKBB7UbmXoSKtNGOuKYYx6VLAE1C5i+5Mw9qe2r3OPnWN/qtRGIdqY0Xqal3AWXUYHH72yjPuKoSzacTuEBQg9hT5oevFZ8sLHPFcde7VmXEn/eeYtzbdeQDnFPu7KVZo5UkhLxsCD94OfbHWqUbSxgx5zGex7VYa2l/s6NxM0pY/u1QZI+mK+dqRcJnQndGhePGbQO2yMR/MvOM//rrOtymqTML5CsbMNik42iq0cTx2ciXAledJVJjbkIvX86YJUmuw5Lquwgdt/wBPWkoct7feTIfYQ20cs+/zwsbsTsU4I7HNJLo9zY28+oROHSVsuO6qff1ratLiA6O0cpHkiPGdwB4//VVW0uWmsxazo/lzZBcnoMfzo9pK7ZNjKvbeC3KusjzRsnC9dvp+FVHhj+zRsvF2W2kdSR3/AAxW42nyaXL9ktla5gnXO6Q8r+fasW3MqXbwsoM7AgZ4xj6/Wt4Surp3GQ6fFK8skUrEKW+6p4BroLLUpo7l7JciUJkuT8uK5mO21CPUGO8tKnLFDnIqzHdOmpSOJCAVG5mHJp1aanfroTY6K0uDpz/ZCWIGXDp3z6+9WLF7i9vLohkiBfKt6ZHSsHQonvdRmubySRguAFJxx9PSukhsltdRk+yN8kxBCHsTXHWSi2upcS3LMdJkiSWT5X4STt+NUY1k/tF4CVBtiWjdTxIjc8ep57U3Uzf7lR7fdCGyCg3dK0LuaKysYVcDzCwWM7eVz3rFaLu2aIpzWUcTPHdReZFIN6S9Gz3z71Sm0+SNRJC5mg6hh1H1FS3LzvcrNNcJKpGFIOFGDyMdj0qJFubWZpbdv3bHJj6r9R6V6OC3tIzkuw1M8c1MrleCaso0F4uWType4Hr/AFpklnJGN33k/vCvbhDsYtiLLz1qZZcdDVbbkcCnAHtW8SS6s7DoxqZZz6ms0PjrUqyVqmwNETE9zThKezGqKy89alDe9UmIti5cd81ILs+gqiGpd4qrgSC4hI+9SfaYecGssI1OCMO2adwNDz4ienNPE47CqSIT2q1FCfwpgTGTd3xTTz3qVIgKlEQ7UxFXy/SnCLParax47VMsakdKVgKItQad9i9av4A6CgjNS4gZUlkDxjNUbizCg5wAOSa1r6+t7NP3jZbsg5JrltR1Ca+yp/dxdkHf61hU5UtSkmV5prWRinmYj5yw74qKUyCNYbW4KKRwu7H41UWFmc47dfap7aJ5bwY+Zdp3c8189iYWnzNnTF6WHWt3BZwtbvJvn5J2Atkn1Nal49tdaKN6JuhT5Gz0z1Hsax4IBaXztIuUYbd392kupLaK4SRJVZUO5152nH9a5nFSkmhstyWMFxpyQwnDMR5f/wBeoL++utPgt4ZIkw7ArKG+X3xVgSpNJBLYoZTgsQPlO08d6ytZvTq9qbeFNkdq4355PPAIPpnj64rSlFylaW35EMty31ydUTYyOHUDviMU24kuBDJbG0mdo23m5Vcqozyc9/oKct5ZyaD5EETmXAwEUhg44zmtXSdSh/stLedlLbPLkXqzk55x1NKT5FdR2AyIMLcuYZwUCgs55JaibRp49MM7zI4yGYAc8ntV2Hw1DFpzm2u9kj/vA7DgL6GmWp1P+yWHA2pmNxxkgcAj86PaXd4PqFjVleCLT2n2YaNRtK8MQKp6XqM020mPczyEluwH8+P6UzTFuDp9159s8t42XiMg4ZSMEfzqTSrhNTsrqyMUdtK6HDdAD/jWDgkmnrqWhZXv4tUMrMwRslGU5GPatO5gklaK4wJjt+cDnI6jHrWZps88McNnKMCJXRyeQw7Y/WpVvbqynIIV4WO5UIxtB7D6UKnzTUU7D5rIsX1vFqESmJlSc/dbOAf9k+lYkTy20xjdWRgcMp4wa12tbbUpfPspzbXf8SP91/qP6ipJrZpcRX0RhnAwsvUH8e4r2sNhfZxtuYylcpiXOD1PrV63vcEBzj3qhJDLaNslXGfusOhpEkB4rtjeJD1NkxRycj5Se46GoXt2U5x+VVIrhounK+ladtcLKOPxU9RXRFpk2KZiI5IyKaU9K1WgBGV6GoXg9sVaQijgqc1IrVIY8cVGVwaYDw2aN3tUfIOMUZx1FMColwelTLcetY8EzMil12MRyKtI/vVJjNVJ6sJcZ4BrIV8fxVMk2OlO4jYS4Gcc5qdJWx/jWRHcE9MCrUcjN1ancDSDnuRUitVISKg3OwAHcnFQTauqjbANx/vEcUm0hWNZ5khj3yMFHqayLzV5HBW2XaP75/pVCSaS5cF2Lv2H/wBanraTN94rEPVzj9Kzcm9h2KMgLOWclmPJJ5NMispbs/IpCeta6WllEMyFrh/TotLPMzx7eEjHRF4FZOn3Hcx5beCBcMd2OiL0/E96zri6k6Q/JngBa0bqMYJNUU3JcBYY98jDCrjrXFXg+iNIsqbbrbHGznjgA9s9/rVu6sopIo7bcFt4z8zL1Y+/tV2a0EUCNcYeQDkZ4qqLh50MBUfLli/oo7f0ryqlKpFq+5opJk8NjLqdvLcRwxRAjYrg4woHU+grHmt1tdFuJIiRbtIPNbPMuD8oX2J5/Crbx3Oo6WzjfHHI5IROFYD19aFtfK0R/MkEqRfejJ/iPQD/AD61nB8ujfXYe5HGFn8PGHTS0JmfLGQ8r0OM1c02EmTUXhnie7eMJFt5wcYYj86ppNqEcuLe2BtLlVBQj7uM9PSrSSHS7iK8MLbZplXy2Od2Rg49MD9cU5t2aXXX/hwSGH7Tp2gW0UluxYytG28cBc5x+PrWxFdb7dmtoZjE6kR5GSB0/Qg1nWLtNrV2ZhvhY/JC/IxngAU2Rv7S+aKV7ezAbyVX+IBiCD7ZH61lOKk7P1v6jSJbfUdUknaARIG4UkKScDv+NXLCG2mvJpEjLZdmDdmGc7x7+vvVH+0jZazCiZaBIkDKRyD3x+OKtIrmUgYJD70QDgxtnHPfP9KicdNrXGMuruKe/jmRCPLYDPTI71PehhJluR2rLinxNIjLlCxwO4rbhxc2a5O5lG0n+tehhsLrcznIz1GDuGQ3Wtez1QhPJuR5kXTJ61mvGUbHSmjj8K9SneBk9TpPscFzbkQuHjP8BPT8e1Yl5pz2bFgCY/UjkfWi3uXhbKMVP1rUi1NpF2yBXHQ5711WjJE6owgeeePrT1ZlYEZBHStZ7fT5udjQk/3Dx+VQPpqE/u7kH03Cj2bWwXLNleiQhJCFY/ka0Gi4rDNjMvRkb6HFalhcvgW9wCrjhWPetI32YmDwioHgz0rTePNQMuKuwjKeEjmm+Wa0mjB7VEYOaLAciBTwxA60pX0oCVBYqyN3qQSEdqjwT7U4KR0paiJ0uGHQVMLyY8LxVUDHWpFzjHXPQDvRdgSmRmO6Qlj6ntVy2s2lAd/kQ9B3NPtLEIFknAz2X0q8TnhapR6sVyEQiJSEG0d/ekEZ/CpxGep6UpPbFOwiuI6jljwOetWSCBmq0uM5zxUtDKM0TSuscalmY4AFaVvYRWEJPDSsPmf/AD2qzY2/lp50g+dhx7CmXj449qhxS1C5g6g5lc/3RU2n6YptHkuB8snUHuOwqWO0Nzcqh+71b6VpXK7k2Lwo4Fc6pJtzY79DnL28nEj+S+yMcAYqsqg2gnmJkVW3yKeuR0rQvbUD5QKqTxFNKmI/vL/hXmYjDato1jIhsZDc2s0wfypSSsag446ULp8MtzBh2PkoN3s3p/WmwxsbSJVGGLFVA75rXMH2BYY4xnAO73NcyoVHzOJfMjOa+hkQpLGY9pZS6KeG7HNPsw9ksdtKo+yWzM5P94MD/ImoJ4AZSSOSckGtJIftGnvGRmRUIX8ulNYZuNkg5tSOaWOe6ilt412ttQkjtmogkmnak7x/xZHPpUGmyAEQsTw4Zfz6V02pWAY7wPxrqw2D91qREp6nLSoRLkjk81oadP5MoVj8j8GmXMJB5FQbSGx2rtjHkehO5vXNvuGQOaznQqeRWjpt2J0+zSn94B8p/vCn3FqRniuvlT1RBkinq5HtTniKE8UzGDQlYLlpZg3B4NSiXFUgTUqscVohFoSZ6GpEmI4PI9KqA5pwY1SYG/DOJUAPWnOtZNvcGNhnpWj52UDKMjvVCGOtR/jUxkVqaVGaAOU8s96UR+lTDHfmjAzUjItgHWkK1IRTcjtSGN2/hWpY2ohAlkHznoD2qCxtwzea3IXpn1rRxk04oQ8ksetSLGKRIwe9TFcAAVYhp4HFAJHbmpQnA9ajkzyAaTArvhs+lMihEtwoxlRyaGHBPartpEIodzfebk1G4EsjhEz7cVkykuxY9e1Xbh9zEdhTLWIPJvP3V6fWolq7DC2hMMOMfvG+8fT2qb7P/e5qdU4J6AcCpjHhQcU7CMO9h3MTj2rLvYtunMv96RR/M/0robtADjHasfUF/dwJ/tFz/L/GuepHdlIj0Wx82VXYfLDkj6mtOa2Lv0qzodvt0/eRy7E/0q48YBAxzVwpJQSBvU5O8tSr9Kdakoy1sX1sOpFZJTyx+NT7NJhczr60MF84TgN86H612cDC+0+Gbj50BP17/rXP3sXnWaTj70R2t/unp+tX/DlySklqT935l+h61UI2k13B7Fe+ttkmMcVjSxlJOK7O8tw65xmudurYhjjtVSgCZQjchgQcMDkH3ro7O5W+t/mx5q/eHr71ze3axJFWLad7aVZU7dR6iiGjBmvPb5BwOaz5IipzitsMlxCJU6NVaaAOpwMGtrEmTjnpSqKlkRgQccios880rDJR0oBpEp7DPIpgOU81agnaM8niqPSnq5HemgNcbW+YdDUm32rNhmKnrxVoS8fepgYGQBTS5zxUXmUbsnrSAk6n1qSOMuwUVEpArRtY9ibm+81CVwJ0QKqqowBxVhIwelMQAnHep1wBgVQiRUxxT9nPJpiMfTJpxbPTigBXGBwaquCRkmpJGO04NVmY49qhgPiTzZVU9Op+lXZZAM1UtPlV3IxuOB9KJZAX46Ck9EBHIST3rQt4wkap36tVS2jMjbyOAePetSKPb1/GiMeoMcibm56CpXXI9qcigCkm4iNNrQDKusFyfwrJvQDPjsigVryYZxnp1rJ5mlyesj5/WsZrSw0dLYw+XYQr6IKc6j0qaMbVC+gpjjn8a3toIpTx74mB61h3URDV00iA8isq8h+UsB14qXEDNs2RmaCU/u5QUb2z3/CqVpI+n34Zlw0TlJB7dDVhl8t8Gm36b9lwOS/yv7kDr+VZtdSkdaCs0QKkFSMg1kXdttdsUnh673I1q55TlPpWpcxZG/HNarVXJOTnt9ueOtVNpXiuimgDBhisua1OCR2qXEdxdNvTbS7HP7p+vsfWtiYY5HQ965sqy9a1dNvRIn2aU8j7pP8AKqi+giSRBIMj7wqm6ZJBGDVyVdrHtULFXODwabGVFJXqKnVtw5pjrtYhuaFyOnSkBIycVHjHFSqcikZc9KYDQeafvPrUJ4NLvNFwME3IHemG9iTlpAPxrg/7QnbrNIf+BGpraR5ZQoyWY4FZ8wHoWnTx3cp2HKJyT2rdjOTWJpUC2tskI6jlj6mtpGAFaoCynB4HNSphScjJqBHwcipN/HuaYibzMdOtG4EYqEMCPencAHFIBJXAXHaoDlwOwpxy30pBjIHYVLGSb9sfHbpUagyuFHUmo5JMtgdBVqyjyTIe/A+lK12BegQAAAcAcVaUHNMRMKKnXGMitCSUDAqG5bjFTZGCap3DDGaljKFw2I5Gz2x+dUrRN17AuONwqzcnMeP7xqPThnUY/QAn/P51k1eSGjpM4HvUTHrTlPHNROeSexrckc/K8duaguIg8RHerIGUP0pjr8ucdKAOcuYjux3zUGzzIJIT1IyPqK1b2Hdll6is4fLKDjvUNDRSs7g2t3FMP4WGfp3rtjiSMHqCK4adNk7pjjJxXV6Ncefp6AnLINppQ7DYyaHDHaao+ScsOtbE6dGFUnXkkVpYkyprXcM4rPeF4XyMjB4I7V0YUEGopLVXBBHWpcRmfDdi4jCScSAfnUchwxp8thhuOMVE0Mu3ruxQMBJng80dDkdKrsxQ8jFOWQ9c0XAtA57Yp3eoVcHvUm71oAR1zUW2pjg0m2lYDxFQc1saGmdQiz25rPEfNa+iri9U+xrGO4Hd2bYArSjO41j2rHArSSTA4rdMC8JNowKcCTjmqqNuFTqcDmmBMCQODQWLcA8UzOeO1Ix7CgALYXimF9qZNVp5xvWFD8xPzH0FEkmSBUtgSpmSRVHVjW7bxhVAHQcVj6aheZpP7vArbT5acUJkwFTj2qBOmTUobkYqhDyflNUJznirjHGRVC4btSYylcHlR7Zo00/6cfZD/MUydssx7dKTTmH21vdD/MVmviGdErfJULHAIzxmlDfKKimPHHrWxJaTlCO9SDBGCO1QxN8n4VMBwPWgDPuo8ZI6VkSRgOCOhreuANprGuEKscdM8UhmZqCbLkH+8oNaHh6fbO8RPDDIqnqfIhbvgjNRafN5F9C3bdistpDOycZ47VQkADlfyrQPIyOtVLlAQHHUVsSVhgZp+3IBFRZwwJ6Gp044HQ0DIZo8jP51QkXYT7VrlQciqVxH19aGBlzqMhu3Q1WeLbyvFXJB1WoFPGKzkNFfJU09Zh3pzpmoHSlcZZ81fWk89fWqmD2pMUcwWPMQvNXrBjHcRkddwFUhVqDggjrWC3A7W2YgYrRjY4rMtTmNT32itJOgreIFtGx0qdSe/Sq0fT8amXk1YibJPPaqVzfAApFye7Ut8zARqCQrZyB3rO6jmonK2g0iS3bMpYntUzP3qCH+OpO4qUNnQadH5dqmep5NaAJ4qvB9xfpU461siCdTlQTUhOce1Rp92pOxoAbI2Kz5z8w/OtB+1Ztx95vpSYFKU5TNRWLldQUeqGpJfun6VBZf8hNf90/yrNbjOkV8pmoJXOCD1Bp0X+pqK4+6a2EXbdsoAatRnjPeqNt91aux0IRDMODWTcja49DWvN1rJuvvGgZl34zAmezGs8NtYEdjWhf/AOpH1/xrPHU1jLcpHbW0omtY3B4YCkccFT0NVNGJOmLn3q3J0rZbEmbJmOQoenUVLBJztqO+6p9aZGTvBpAaQ+YfSoJUzk1Knf6UjfcNUBiXSFTVHcUc+hrTvvu/hWY4+UVEhkoIYUx0psR+apm6VBRUZcUYFSuKZgUAf//Z"
      },
      "key": "   1"
    }
  ]
}

curl을 이용해 모델 서버로 JSON을 보냅니다.

$ curl "https://{projectName}--flower-sample.srv.kpb3r.mlxp.ncloud.com/v1/models/flower-sample:predict" -XPOST -d@./input.json

{
    "predictions": [
        {
            "scores": [0.999114931, 9.20987877e-05, 0.000136786475, 0.000337258185, 0.000300532876, 1.84813962e-05],
            "prediction": 0,
            "key": "   1"
        }
    ]
}

요청을 보내는 URL은 세 가지로 구성되어 있습니다.

  1. IngressRoute URL https://{projectName}--{ingressrouteName}.srv.kpb3r.mlxp.ncloud.com
  2. serving ingress port 443
  3. URL의 path /v1/models/flower-sample:predict
    • URL 경로는 모델 서버 Runtime API에 기반합니다.
    • KServe Runtime을 사용할 경우 ServerRuntimes는 /v1/models/ API를 사용합니다.
    • gRPC를 사용하는 경우 path는 사용하지 않습니다.
주의

자체 이미지를 사용할 경우 해당 ServingRuntime에서 정의한 API를 사용해야 합니다.

Using gRPC

gRPC를 제공하는 모델 서버라면 InferenceService YAML을 변경하여 gRPC 포트를 노출할 수 있습니다.
KServe의 내장된 모델 서버들은 gRPC를 위해 9000 container 포트를 사용합니다.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: flower-grpc
  namespace: p-{projectName}
spec:
  predictor:
    model:
      modelFormat:
        name: tensorflow
      storageUri: "gs://kfserving-examples/models/tensorflow/flowers" #Mount Path 정보 입력
      ports:
        - containerPort: 9000
          name: h2c
          protocol: TCP
주의

port 이름은 반드시 h2c로 설정해야 합니다.

외부 엔드포인트를 통해 모델 서버에 접근하기 위해 아래 IngressRoute를 생성합니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: flower-grpc
spec:
  httpRoutes:
  - route:
    - inferenceServiceName: flower-grpc
      weight: 100

YAML을 적용한 후 gRPC 클라이언트를 사용해 요청을 보냅니다.

import json
import base64
import grpc

from tensorflow.core.framework import tensor_pb2
from tensorflow.python.framework import tensor_util
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc

if __name__ == '__main__':
    host = "{projectName}--flower-grpc.srv.kpb3r.mlxp.ncloud.com"
    port = 80  # 443 for TLS
    channel = grpc.insecure_channel(target=f"{host}:{port}")
    stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

    with open("./input.json", "r") as json_file:
        data = json.load(json_file)
    image = data['instances'][0]['image_bytes']['b64']
    key = data['instances'][0]['key']

    # Call classification model to make prediction
    request = predict_pb2.PredictRequest()
    request.model_spec.name = "flower-grpc"  # Same as InferenceService name field
    request.model_spec.signature_name = "serving_default"  # Always serving_default
    image = base64.b64decode(image)
    request.inputs['image_bytes'].CopyFrom(tensor_util.make_tensor_proto(image, shape=[1]))
    request.inputs['key'].CopyFrom(tensor_util.make_tensor_proto(key, shape=[1]))

    result = stub.Predict(request, 10.0)
    print(result)

Predictor Runtimes 정보

ML expert Platform이 지원하는 모델 포맷별 기본 Runtime은 다음과 같습니다. InferenceService 작성시 모델 포맷에 따라 적절한 Runtime이 자동으로 선택됩니다.

모델 포맷 기본 Runtime
Tensorflow kserve-tensorflow-serving
PyTorch kserve-torchserve
MLFlow kserve-mlserver
LightGBM kserve-lgbserver
Paddle kserve-paddleserver
SKLearn kserve-sklearnserver
XGBoost kserve-xgbserver
HuggingFace kserve-huggingface
참고
  • ML expert Platform에서 제공하는 Runtime 외 특정 프로젝트 전용 Serving 환경이 필요한 경우 ServingRuntime을 직접 정의해 사용할 수 있습니다.
  • ServingRuntime은 KServe에서 모델을 서빙할 컨테이너 이미지와 실행 방식을 정의하는 네임스페이스 범위의 리소스입니다. 자세한 내용은 Serving Runtimes을 참고하시기 바랍니다.

KServe의 더 많은 예제는 KServe 공식 예제를 참고하시기 바랍니다.

IngressRoute 사용

IngressRoute는 외부에서 접근 가능한 엔드포인트를 제공하는 기능입니다. 해당 엔드포인트로 들어오는 요청을 여러 InferenceService에 적절하게 분산하거나, ML expert Platform 외부로 요청을 라우팅할 수 있습니다.

InferenceService 간 트래픽 분배

아래는 IngressRoute를 이용해 다수의 InferenceService로 트래픽을 분산하는 방법을 안내합니다. 다음과 같은 시나리오에서 유용합니다.

  • 다른 GPU 장비에 동일한 모델을 배포하는 경우
  • 다른 버전의 모델 애플리케이션을 카나리 배포하는 경우

아래는 a100-model로 트래픽을 30%, h100-model로 트래픽을 70%씩 분산하는 예시입니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-ingress
  namespace: p-{projectName}
spec:
  httpRoutes:
  - route:
    - inferenceServiceName: a100-model
      weight: 30
    - inferenceServiceName: h100-model
      weight: 70

ML expert Platform 내부로 트래픽 라우팅

{service}.{namespace}.svc.cluster.local와 같은 k8s service 도메인으로 라우팅 가능합니다.
아래는 a100-model로 트래픽을 90%, 외부 서비스인 google.com으로 트래픽을 10%씩 분산하는 예시입니다.
internalServiceURL을 설정할 때 반드시 scheme 또는 port 정보를 입력해야 하며, k8s service 도메인의 경우 svc.cluster.local을 포함한 도메인을 사용해야 합니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-ingress
  namespace: p-{projectName}
spec:
  httpRoutes:
  - route:
    - internalServiceURL: a100-model.p-{projectName}.svc.cluster.local:80
      weight: 90
    - externalServiceURL: https://google.com
      weight: 10

ML expert Platform 외부로 트래픽 라우팅

아래는 IngressRoute를 이용해 ML expert Platform 외부로 트래픽을 라우팅하는 방법입니다. ML expert Platform에 배포되어 있지 않은 외부 모델로 트래픽을 분산하고 싶은 경우 유용합니다.
아래는 a100-model로 트래픽을 90%, 외부 서비스인 google.com으로 트래픽을 10% 씩 분산하는 예시입니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-ingress
  namespace: p-{projectName}
spec:
  httpRoutes:
  - route:
    - inferenceServiceName: a100-model
      weight: 90
    - externalServiceURL: https://google.com
      weight: 10

조건 기반 트래픽 라우팅

여기서는 match 스펙을 활용해 조건에 따라 라우팅하는 방법을 소개합니다. HTTP 트래픽의 요청 조건에 따라 트래픽을 라우팅할 때 유용합니다.
match 스펙에 대한 자세한 설명은 HTTPMatchRequest를 참고하시기 바랍니다.

아래는 model header값에 따라 트래픽을 라우팅하는 예시입니다. a100인 경우 모든 트래픽은 a100-model로 가고 h100인 경우 90%의 트래픽은 h100-model-v1, 10%의 트래픽은 h100-model-v2로 분산됩니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-ingress
  namespace: p-{projectName}
spec:
  httpRoutes:
  - match:
    - headers:
        model: a100
    route:
    - inferenceServiceName: a100-model
  - match:
    - headers:
        model: h100
    route:
    - inferenceServiceName: h100-model-v1
      weight: 90
    - inferenceServiceName: h100-model-v2
      weight: 10

아래는 URL 경로에 따른 트래픽 라우팅 예시입니다. /api/v1 으로 오는 요청은 model-v1 으로, /api/v2 로 오는 요청은 model-v2 로 라우팅됩니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-ingress
  namespace: p-{projectName}
spec:
  httpRoutes:
  - match:
    - uri:
        prefix: /api/v1
    route:
    - inferenceServiceName: model-v1
  - match:
    - uri:
        prefix: /api/v2
    route:
    - inferenceServiceName: model-v2

트래픽 미러링

IngressRoute의 .spec.httpRoutes[].Mirrors 스펙을 활용해 IngressRoute로 들어온 트래픽을 다른 곳으로 복제 전송할 수 있습니다. 실서비스의 트래픽을 받기 전 테스트할 때 유용합니다.

internalServiceURL을 사용한 IngressRoute 미러링

아래는 model-a로 들어오는 트래픽을 model-b로 미러링하는 예시입니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-a
  namespace: p-{project}
spec:
  httpRoutes:
  - route:
    - internalServiceURL: http://model-a.p-{project}.svc.cluster.local
    mirrors:
    - destination:
        externalServiceURL: http://{project}--model-b.srv.kpb3r.mlxp.ncloud.com

IngressRoute를 생성하면 {project}--model-a.srv.kpb3r.mlxp.ncloud.com 엔드포인트가 생성됩니다. 해당 엔드포인트로 요청이 들어오면 Host 헤더에 기존 엔드포인트에 -shadow가 붙은 값인 {project}--model-a.srv.kpb3r.mlxp.ncloud.com-shadow가 설정되어 Mirror destination으로 전달합니다.

이때 model-b에서는 아래처럼 hosts를 추가해야 Mirror 트래픽을 받을 수 있습니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-b
  namespace: p-{project}
spec:
  hosts:
  - {project}--model-a.srv.kpb3r.mlxp.ncloud.com-shadow
  httpRoutes:
  - route:
    - internalServiceURL: http://model-b.p-{project}.svc.cluster.local

inferenceServiceName을 사용한 IngressRoute 미러링

.spec.httpRoutes[].headers에 Host 헤더가 설정된다면 해당 헤더에 -shadow Postfix가 붙은 Host 헤더가 Mirror 트래픽에 설정됩니다. kserve InferenceService 에 대한 IngressRoute 를 배포할 경우 Host 헤더에 항상 InfereceService 의 .status.URL 의 Host 부분이 설정됩니다.
아래는 kserve InferenceService 형태로 배포된 model-a 에 대한 IngressRoute 예시입니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-a
  namespace: p-{project}
spec:
  httpRoutes:
  - route:
    - inferenceServiceName: model-a
    mirrors:
    - destination:
        externalServiceURL: http://{project}--model-b.srv.kpb3r.mlxp.ncloud.com

아래는 model-b IngressRoute에서 Mirror 트래픽을 받기 위한 예시입니다. model-a InferenceService의 .status.URL에 대한 -shadow postfix 가 붙은 Host 값을 설정해야 합니다.

apiVersion: endpoint.mlx.navercorp.com/v1alpha1
kind: IngressRoute
metadata:
  name: model-b
  namespace: p-{project}
spec:
  hosts:
  # transformer 가 정의되어 있다면 predictor 대신 transformer 로 변경
  - model-a-predictor.p-{project}.svc.cluster.local-shadow
  httpRoutes:
  - route:
    - internalServiceURL: http://model-b.p-{project}.svc.cluster.local
사업자등록번호: 129-86-31394 통신판매업신고번호:제2009-경기성남-0510호
대표이사: 김유원 주소: 경기도 성남시 분당구 불정로 6 네이버 그린팩토리 17~26층, 13561 고객지원 대표전화: 1544-5876
© NAVER Cloud Corp. All Rights Reserved.