The Data (error) Generating Process | Emily Riederer

Previously streamed live at MDS Fest

The Data (error) Generating Process, a talk by Emily Riederer, Senior Manager of Analytics, Capital One

Talk Description

Statisticians often approach probabilistic modeling by first understanding the conceptual data generating process. However, when validating messy real-world data, the technical aspects of the data generating process is largely ignored. In this talk, I will argue the case for developing more semantically meaningful and well-curated data tests by incorporating both conceptual and technical aspects of "how the data gets made". To illustrate these concepts, we will explore the NYC subway rides open dataset to see how the simple act of reasoning about real-world events their collection through ETL processes can help craft far more sensitive and expressive data quality checks. I will also illustrate instrumenting such checks based on new features in the dbt-utils package (with the grouping functionality that I contributed). This talk should be of interest to analytics engineers looking for frameworks to improve their data quality. Audience members should leave this talk with a clear framework in mind for ideating better tests for their own pipelines.