Gold Sponsors

  • Don Bosco Wilrijk
  • Inuits
  • VanTosh

Bronze Sponsors

  • IT Partners
  • MySQL
  • Icinga

Scheduling data pipelines (Bert Desmet)

Scheduling data pipelines (Bert Desmet)

The back-end of big analytical projects are often split in multiple small tasks. Most 'enterprise' tools include their own scheduler, but if you want to combine your ETL flow with your spark jobs with specific python or R scripts, a more specialized scheduler will be needed. In this session I will explain multiple scheduling concepts (DAG, triggers, Branching, parameters and ‘batching’) with Airflow, AirBNB’s own scheduling solutions.

Bert Desmet

Bert is a data consultant dat Deloitte's 'Analytics and Information Management' practice.^ In his current roll he acts as an architect on some of Deloitte Belgiums bigger data driven projects.^ Beforing taking up his current roll, Bert was active as a system engineer for Deloitte's Analytics Platform.^ Next to that Bert has a general interest in Open Source, politics, hiking, biking an beer.^

Twitter : @bdesmet_