Airflow | search
Click here to view Airflow news from 60+ newspapers.
Bookmark or Share- airflow - Is there a way to configure different 'retries' for tasks in ...I have a DAG with many sub-tasks in it. In the middle of the DAG, there is a validation task and based on the result/return code from the task, i want to take two different paths. If success, one r...
- How to add manual tasks in an Apache Airflow DagI'm using Apache Airflow to manage the data processing pipeline. In the middle of the pipeline, some data need to be reviewed before the next-step processing. e.g. ... -> task1 -> human review -> task2 -> ... where task1 and task2 are data processing task. When task1 finished, the generated data by task1 needs to be reviewed by human.
- Writing to Airflow Logs - Stack OverflowAirflow's preconfigured logger is returned only for a process started by airflow and all threads started by the main process. It doesn't work for a new process(a new default logger is created for a new process). But it does work for a new thread started by the same process. –
- python - How to control the parallelism or concurrency of an Airflow ...Options that are specified across an entire Airflow setup: core.parallelism: maximum number of tasks running across an entire Airflow installation; core.dag_concurrency: max number of tasks that can be running per DAG (across multiple DAG runs) core.non_pooled_task_slot_count: number of task slots allocated to tasks not running in a pool
- How to rename a DAG in Apache Airflow - Stack OverflowIm simply doing a rename of the job, not change to the business logic etc. When this is deployed into Airflow, it is automatically picked up and registered as a new job, so there are now 2 jobs visible in the DAG view. hello_world; yet_another_hello_world
More
