Airflow未运行SparkSubmitOperator命令

0 人关注

我有如下的airflow dag python脚本，它没有给出任何错误，但它没有运行底层的SparkSubmitOperator命令。如何解决这个问题来运行Spark-submit命令。我想执行的命令如下。

spark-submit --master local[*] --executor-memory 1G --driver-memory 1G  --class com.agreeya.iot.IOTDataParser /root/iot/IOTParser/IOTDataParser.jar
Airflow Python DAG脚本
from datetime import datetime, timedelta
import pendulum
from airflow import DAG
from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
from airflow.models import Variable
local_tz = pendulum.timezone("Asia/Kolkata")
default_args = {
    'owner': 'agreeya',
    'depends_on_past': False,
    'start_date': datetime(2020, 10, 10, tzinfo=local_tz),
    'email': ['maitreya.bhatnagar@agreeya.com'],
    'email_on_failure': True,
    'email_on_retry': True,
    'retries': 2,
    'retry_delay': timedelta(minutes=5)
dag = DAG(dag_id='scaqmd_weather_monitoritoring_app',
          default_args=default_args,
          catchup=False,
          schedule_interval="0 * * * *")
iot_weather_ingestion = SparkSubmitOperator(task_id='iot_weather_ingestion',
                                              conn_id='spark_default',
                                              conf={'master':'local[*]'},
                                              application='/root/iot/IOTParser/IOTDataParser.jar',
                                              java_class="com.agreeya.iot.IOTDataParser",
                                              total_executor_cores=1,
                                              executor_cores=1,
                                              executor_memory='1g',
                                              driver_memory='1g',
                                              name='iot_weather_ingestion',
                                              execution_timeout=timedelta(minutes=10),