CLI (v2) 命令作业 YAML 架构 - Azure Machine Learning

YAML 语法


   compute

要在其上执行作业的计算目标的名称。这可以是对工作区中现有计算的引用（使用


   azureml:<compute_name>

语法），也可以是对


   local

的引用，以指定本地执行。注意：管道中的作业不支持将


   local

作为


   compute


   local


   resources.instance_count

用于作业的节点数。


   resources.instance_type

用于作业的实例类型。适用于在启用了 Azure Arc 的 Kubernetes 计算上运行的作业（其中


   compute

字段中指定的计算目标是


   type: kubernentes

）。如果省略，这将默认为 Kubernetes 群集的默认实例类型。有关更多信息，请参阅创建和选择 Kubernetes 实例类型。


   resources.shm_size

Docker 容器的共享内存块的大小。它应采用以下格式：


   <number><unit>

，其中数字必须大于 0，单位可以是

（字节）、

（千字节）、

（兆字节）或

（千兆字节）之一。


   limits.timeout

允许作业运行的最长时间（秒）。达到此限制后，系统会取消该作业。作业输入的类型。为指向单个文件源的输入数据指定


   uri_file

，或为指向文件夹源的输入数据指定


   uri_folder

。


   uri_file


   uri_folder


   mlflow_model


   custom_model


   uri_folder

用作输入的数据的路径。这可以通过几种方式指定：

- 数据源文件或文件夹的本地路径，例如


   path: ./iris.csv

。数据将在作业提交期间上传。

- 要用作输入的文件或文件夹的云路径的 URI。支持的 URI 类型为


   azureml

、


   https

、


   wasbs

、


   abfss

、

adl

。有关如何使用


   azureml://

URI 格式的详细信息，请参阅核心 YAML 语法。

- 要用作输入的现有已注册的 Azure 机器学习数据资产。若要引用已注册的数据资产，请使用


   azureml:<data_name>:<data_version>

语法或


   azureml:<data_name>@latest

（引用该数据资产的最新版本），例如


   path: azureml:cifar10-data:1

或


   path: azureml:cifar10-data@latest

。将数据传送到计算目标的模式。

对于只读装载 (


   ro_mount

)，数据将用作装载路径。文件夹将装载为文件夹，文件将装载为文件。 Azure 机器学习会将输入解析为装载路径。

对于


   download

模式，数据将下载到计算目标。 Azure 机器学习会将输入解析为下载的路径。

如果你只想要数据工件的存储位置的 URL 而不是挂载或下载数据本身，则可以使用


   direct

模式。这将传入存储位置的 URL 作为作业输入。请注意，在这种情况下，你全权负责处理凭据以访问存储。


   ro_mount


   download


   direct


   ro_mount

作业输出的类型。对于默认的


   uri_folder

类型，输出将对应一个文件夹。


   uri_folder


   mlflow_model


   custom_model


   uri_folder

输出文件传送到目标存储的模式。对于读写装载模式 (


   rw_mount

)，输出目录是装载的目录。对于上传模式，写入的文件将在作业结束时上传。


   rw_mount


   upload


   rw_mount

UserIdentityConfiguration

YAML：hello world


   $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: echo "hello world"
environment:
  image: library/python:latest
compute: azureml:cpu-cluster
   
    YAML：显示名称、试验名称、说明和标记
   
   
    $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: echo "hello world"
environment:
  image: library/python:latest
compute: azureml:cpu-cluster
tags:
  hello: world
display_name: hello-world-example
experiment_name: hello-world-example
description: |
  # Azure Machine Learning "hello world" job
  This is a "hello world" job running in the cloud via Azure Machine Learning!
  ## Description
  Markdown is supported in the studio for job descriptions! You can edit the description there or via CLI.
    
     YAML：环境变量
    
    
     $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: echo $hello_env_var
environment:
  image: library/python:latest
compute: azureml:cpu-cluster
environment_variables:
  hello_env_var: "hello world"
     
      YAML：源代码
     
     
      $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: ls
code: src
environment:
  image: library/python:latest
compute: azureml:cpu-cluster
      
       YAML：文字输入
      
      
       $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: |
  echo ${{inputs.hello_string}}
  echo ${{inputs.hello_number}}
environment:
  image: library/python:latest
inputs:
  hello_string: "hello world"
  hello_number: 42
compute: azureml:cpu-cluster
       
        YAML：写入默认输出
       
       
        $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: echo "hello world" > ./outputs/helloworld.txt
environment:
  image: library/python:latest
compute: azureml:cpu-cluster
        
         YAML：写入命名数据输出
        
        
         $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: echo "hello world" > ${{outputs.hello_output}}/helloworld.txt
outputs:
  hello_output:
environment:
  image: python
compute: azureml:cpu-cluster
         
          YAML：数据存储 URI 文件输入
         
         
          $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: |
  echo "--iris-csv: ${{inputs.iris_csv}}"
  python hello-iris.py --iris-csv ${{inputs.iris_csv}}
code: src
inputs:
  iris_csv:
    type: uri_file 
    path: azureml://datastores/workspaceblobstore/paths/example-data/iris.csv
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
compute: azureml:cpu-cluster
          
           YAML：数据存储 URI 文件夹输入
          
          
           $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: |
  ls ${{inputs.data_dir}}
  echo "--iris-csv: ${{inputs.data_dir}}/iris.csv"
  python hello-iris.py --iris-csv ${{inputs.data_dir}}/iris.csv
code: src
inputs:
  data_dir:
    type: uri_folder 
    path: azureml://datastores/workspaceblobstore/paths/example-data/
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
compute: azureml:cpu-cluster
           
            YAML：URI 文件输入
           
           
            $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: |
  echo "--iris-csv: ${{inputs.iris_csv}}"
  python hello-iris.py --iris-csv ${{inputs.iris_csv}}
code: src
inputs:
  iris_csv:
    type: uri_file 
    path: https://azuremlexamples.blob.core.windows.net/datasets/iris.csv
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
compute: azureml:cpu-cluster
            
             YAML：URI 文件夹输入
            
            
             $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: |
  ls ${{inputs.data_dir}}
  echo "--iris-csv: ${{inputs.data_dir}}/iris.csv"
  python hello-iris.py --iris-csv ${{inputs.data_dir}}/iris.csv
code: src
inputs:
  data_dir:
    type: uri_folder 
    path: wasbs://datasets@azuremlexamples.blob.core.windows.net/
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
compute: azureml:cpu-cluster
             
              YAML：通过 papermill 的笔记本
             
             
              $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: |
  pip install ipykernel papermill
  papermill hello-notebook.ipynb outputs/out.ipynb -k python
code: src
environment:
  image: library/python:latest
compute: azureml:cpu-cluster
              
               YAML：基本 Python 模型训练
              
              
               $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: src
command: >-
  python main.py 
  --iris-csv ${{inputs.iris_csv}}
  --C ${{inputs.C}}
  --kernel ${{inputs.kernel}}
  --coef0 ${{inputs.coef0}}
inputs:
  iris_csv: 
    type: uri_file
    path: wasbs://datasets@azuremlexamples.blob.core.windows.net/iris.csv
  C: 0.8
  kernel: "rbf"
  coef0: 0.1
environment: azureml:AzureML-sklearn-0.24-ubuntu18.04-py37-cpu@latest
compute: azureml:cpu-cluster
display_name: sklearn-iris-example
experiment_name: sklearn-iris-example
description: Train a scikit-learn SVM on the Iris dataset.
               
                YAML：使用本地 Docker 生成上下文进行基本 R 模型训练
               
               
                $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: >
  Rscript train.R 
  --data_folder ${{inputs.iris}}
code: src
inputs:
  iris: 
    type: uri_file
    path: https://azuremlexamples.blob.core.windows.net/datasets/iris.csv
environment:
  build:
    path: docker-context
compute: azureml:cpu-cluster
display_name: r-iris-example
experiment_name: r-iris-example
description: Train an R model on the Iris dataset.
                
                 YAML：分布式 PyTorch
                
                
                 $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: src
command: >-
  python train.py 
  --epochs ${{inputs.epochs}}
  --learning-rate ${{inputs.learning_rate}}
  --data-dir ${{inputs.cifar}}
inputs:
  epochs: 1
  learning_rate: 0.2
  cifar: 
     type: uri_folder
     path: azureml:cifar-10-example@latest
environment: azureml:AzureML-pytorch-1.9-ubuntu18.04-py37-cuda11-gpu@latest
compute: azureml:gpu-cluster
distribution:
  type: pytorch 
  process_count_per_instance: 1
resources:
  instance_count: 2
display_name: pytorch-cifar-distributed-example
experiment_name: pytorch-cifar-distributed-example
description: Train a basic convolutional neural network (CNN) with PyTorch on the CIFAR-10 dataset, distributed via PyTorch.
                 
                  YAML：分布式 TensorFlow
                 
                 
                  $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: src
command: >-
  python train.py 
  --epochs ${{inputs.epochs}}
  --model-dir ${{inputs.model_dir}}
inputs:
  epochs: 1
  model_dir: outputs/keras-model
environment: azureml:AzureML-tensorflow-2.4-ubuntu18.04-py37-cuda11-gpu@latest
compute: azureml:gpu-cluster
resources:
  instance_count: 2
distribution:
  type: tensorflow
  worker_count: 2
display_name: tensorflow-mnist-distributed-example
experiment_name: tensorflow-mnist-distributed-example
description: Train a basic neural network with TensorFlow on the MNIST dataset, distributed via TensorFlow.
                  
                   YAML：分布式 MPI
                  
                  
                   $schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: src
command: >-
  python train.py
  --epochs ${{inputs.epochs}}
inputs:
  epochs: 1
environment: azureml:AzureML-tensorflow-2.7-ubuntu20.04-py38-cuda11-gpu@latest
compute: azureml:gpu-cluster
resources:
  instance_count: 2
distribution:
  type: mpi
  process_count_per_instance: 1
display_name: tensorflow-mnist-distributed-horovod-example
experiment_name: tensorflow-mnist-distributed-horovod-example
description: Train a basic neural network with TensorFlow on the MNIST dataset, distributed via Horovod.
                   
                    
                     安装并使用 CLI (v2)