Reversed Upstream/downstream Relationships When Generating Multiple Tasks In Airflow

The original code related to this question can be found here. I'm confused by up both bitshift operators and set_upstream/set_downstream methods are working within a task loop tha

Solution 1:

The tree view in Airflow is "backwards" to how you (and I!) first thought about it. In your first screenshot it is showing that "clear_tables" must be run before the "AAAG5608078M2" run task. And the DAG status depends upon each of the id worker tasks. So instead of a task order, it's a tree of the status chain. If that makes any sense at all.

(This might seem strange at first, but it's because a DAG can branch out and branch back in.)

You might have better luck looking at the Graph view for your dag. This one has arrows and shows the execution order in a more intuitive way. (Though I do now find the tree view useful. It's just less clear to start with)

Solution 2:

Looking through your other code, it seems get_id_creds is your task and you're trying to loop through it, which is creating some weird interaction.

A pattern that will work is:

clear_tables = MyOperator()

for uid in uid_list:
  my_task = MyOperator(task_id=uid)
  clear_tables >> my_task