Session

Nate Dotzlaf

Nate Dotzlaf

Executive Director, IS – Enterprise Analytics

Indiana University Health – Information Services

Data Pipeline Modernization at Scale

Healthcare organizations aggregate petabytes of data which help drive life-changing health and business decisions. In this presentation, we’ll show you how Indiana’s largest healthcare institution went from using proprietary tools that were hard to maintain and nearly impossible to scale — to implementing a new world of cloud native and open source tooling, including Airflow, Spark, Delta Lakes (delta.io) and Terraform. We’ll showcase how making a big change to a data pipeline infrastructure can:

  • simplify and automate workflow processes;
  • enable organizations to import existing and new data sources quickly, efficiently and at scale;
  • keep costs down in relation to the number and volume of data sources;
  • hire and onboard new developers — who have experience working with the open source tools — quickly, eliminating the tech learning curve that existed with the proprietary tools.