How Clean Is Clean Enough? Actionable Steps for Making Data AI-Ready

Improving data quality is a matter of financial preservation, as Gartner reports that poor data quality costs businesses an average of $12.9 million annually. Compounding errors made by AI models fed dirty data can affect performance, increase business risk, and leave organizations with a considerable, and frankly avoidable, price to pay.

Data teams are well aware of the clean-up job they’re usually tasked to do and of its immediate importance. But essential questions around data cleansing still stand: How clean is clean enough? What parameters should data teams use to define data quality? And — most importantly — how do you even clean data?

First and foremost, data clean-up must be an all-hands task that involves early intervention to ensure high quality from the source: Data producers. To build a smooth data collection process that ensures high-quality data, data teams should:

• Establish clear data guidelines for quality and cleanliness
• Democratize data governance roles, processes, and responsibilities.
• Encourage a top-down culture of data cleanliness with stakeholder buy-in.
• Utilize data tools that enable observability to mitigate resource strain.

1. Establish Clear Guidelines for Data Quality and Cleanliness

To encourage all data producers to take ownership of data quality, it’s essential to create clear — and well-understood — guidelines for data cleanliness. These guidelines should address the essential pillars of data quality assurance.

Once those guidelines are established, data teams should leverage the appropriate tooling to help continuously monitor data and run tests regularly to ensure data quality. These tests should not only vet for data freshness but also an organization’s ability to meet any existing service-level agreements.

To help data teams get started on their guidelines, we’ve created three tiers of framework you can build your policies and processes around. These separate tiers represent the three core maturity stages of your data program.

Novice

A novice team has some basic assumptions developed about data and some basic data quality checks in place (e.g. assertion tests) to measure how well the underlying quality of the data follows developed assumptions about the data. In the event of failed data quality checks, the team may be notified or may have to manually check for test failures. SLAs for data quality fixes may not be well defined, and end users do not know the state of data quality.

• Completeness Checks: All fields are populated (e.g. not_null assertion test in dbt).
• Validity Checks: Verify data conforms to predefined formats and ranges (e.g. accepted_values tests in dbt, or expect_column_values_to_be_between test in dbt_expectations package).
• Uniqueness Checks: Confirm each entry is unique where required, preventing duplicates (e.g. unique assertion test in dbt).
• Consistency Checks: Check for data that contradicts in different parts of the database (e.g. relationships test in dbt).
• Timeliness Checks: Verify data is updated and available within expected time frames (e.g. source freshness checks in dbt).

Intermediate

A team in the intermediate tier likely has additional layers of data quality checks, beyond assertion tests, in place to proactively catch data quality issues before they begin. This should include things like development testing (e.g. CI checks) and unit tests. These teams also have more robust data quality alerting and management procedures in place (e.g. a fire team or on-call rotation). The data team tracks data quality metrics and metadata and uses them to improve their data quality program.

• Accuracy Checks: Validate data against a known, reliable source to ensure correctness.
• Stewardship Checks: Enforce required fields and surface assets that may not have appropriate owners set.
• Continuous Integration Checks: Test changes to data transformation code before pushing into production.
• Unit Tests: Evaluate the logic of data transformations, rather than the underlying data, to confirm the accuracy and appropriate business context (e.g. Unit Tests coming to dbt 1.8).
• Usability Checks: Evaluate how easily data can be used and understood by end-users

Advanced/Expert

Advanced teams expose their data quality score to data producers and consumers, and use it as a key metric to optimize towards. These teams likely have a high Data Trust score and use metadata as a key asset to improve data quality. In addition, SLAs for data downtime are clearly defined and met.

• Dimensional Scoring: Apply multi-dimensional analysis for detailed data quality assessment, covering Accuracy, Reliability, Stewardship, and Usability.
• Data Lineage Completeness: Ensure different assets have both downstream and upstream owners.
• Data Classification: Data classification techniques to detect PII, PHI, etc.

Data teams should ensure these guidelines are directly influenced by the metadata and context of regularly collected data. This may entail creating different guidelines for different types or sources of data. Defining guidelines with this level of specificity and clarity is essential as it will mitigate the risk of AI hallucinations, which can cause a ripple effect of errors cascading throughout an organization and into its services.

2. Democratize Data Governance Roles, Processes, and Responsibilities

Addressing dirty, inaccurate, or incomplete data requires a transformational shift in data collection processes. This includes assigning data governance owners and establishing clear expectations for their role in an organization’s larger data governance framework. In addition to a team of skilled data analysts, organizations should assign domain experts to take the data reins in specific functional units.

It’s essential data teams communicate this is not intended to take away decision-making from those on the business side. Mandating a careful eye toward data cleanliness can support the work of business teams by enabling faster productivity, reducing risky (and costly errors), and enhancing performance.

In practical terms, data teams can institute efficient data governance processes by creating regular data reports on a consistent cadence to keep early tabs on data cleanliness at the collection stage. These reports can then be shared with department leaders and other stakeholders to ensure steady performance. Additionally, data teams can utilize the following structures and processes to help facilitate data democratization:

• Data Stewardship Committees: Establish cross-functional data stewardship committees composed of representatives from various departments across the organization. These committees collaborate to define data standards, policies, and best practices, ensuring that data governance decisions reflect the needs and perspectives of diverse stakeholders.
• Data Catalogs: Develop community-driven data catalogs where users can contribute metadata, annotations, and usage feedback for data assets. By crowdsourcing metadata management, organizations can leverage the collective knowledge and expertise of users to enhance the discoverability and understanding of data assets.
• Self-Service Data Governance Tools: Implement self-service data governance tools that enable business users to contribute metadata, define data quality rules, and request access to data assets. These tools empower users to take ownership of data governance tasks within their respective domains, reducing reliance on centralized governance teams.

More specifically, data collaboration tools can help enable better data governance processes. These solutions bring engineers and producers together during data collection, which helps provide more context around existing data and connect data cleanliness metrics to business impacts.

3. Encourage A Top-Down Culture of Data Cleanliness with Stakeholder Buy-in

It’s essential to get buy-in from stakeholders and leaders and establish a culture that values and understands the importance of data quality, from the top down. Similarly to democratizing data governance, instituting a data-driven culture helps employees understand how data quality specifically contributes to achieving their strategic objectives and aligning with broader organizational goals.

That’s much easier said than done. Here are some practical tips data teams can leverage to get buy-in from the top for data clean-up:

• Focus on the risks. The technical aspects and benefits of clean data aren’t as compelling to C-levels as the risks of using rogue data to build AI models. Most practically, data teams can assess the financial impact of compliance risks and costs that come with operating (and iterating) on dirty data — as well as its impact on brand reputation. Illuminating the hefty fines that come with regulatory violations can be an especially effective method of speaking to executives’ priorities.
• Put the cost of dirty data in concrete terms. It’s essential to translate risk into financial impact for stakeholders; that way, you’ll be speaking their language. Establish financial models and find reputable resources from accredited research firms (like Gartner) that can project with as much accuracy as possible how AI models built on rogue data can incur costs due to poor performance.
• Emphasize the impact on products or features. Rogue data directly impacts the efficiency and effectiveness of an organization’s products and features. Focusing on how dirty data creates a significant negative impact on the services your organization is building (such as customer churn due to poor performance) will paint a clear picture for C-levels of both the long-term and short-term risks of failing to invest in data clean-up.

4. Utilize Data Tools that Enable Observability to Mitigate Resource Strain

Resource strain is one of the foremost challenges to effective data clean-up. Too often, data teams are incorporated as an afterthought or very late in an organization’s production processes. As such, there are already multiple existing data sources that they must continuously monitor and vet for cleanliness. 

These sources can be too numerous and disparate for human eyes to manually track alone. As such, organizations should invest early in data management and governance tools that enable the observability data teams need to ensure data quality. These observability tools also naturally help implement the consistency across departments and sources that data clean-up processes need to be successful, scalable, and sustainable.

More tools are not necessarily better. What’s essential here is for data teams to identify solutions that can consolidate data needs and prevent data asset sprawl. The right tool can also enable:

• Data Quality Monitoring: Observability tools can help data teams monitor the quality of data flowing through systems in real time. Better monitoring means faster detection of anomalies, errors, or inconsistencies in data streams — enabling faster data clean-up.
• Metadata Management: These tools can also offer a view into metadata associated with data streams, such as timestamps, source information, and data formats. Metadata here is essential for lineage tracking, data provenance, and data discovery — all of which aid in clean-up as well as ensuring proper governance.
• Data Lineage and Impact Analysis: By capturing information about data flows and dependencies, observability tools can facilitate data lineage and impact analysis. This enables organizations to enhance governance processes with a better understanding of systems, a sharp eye on potential bottlenecks or points of failure, and accurate assessments of changes necessary to facilitate better downstream processes.
• Data Archiving and Purging: Based on insights gained from observability tools, organizations can develop strategies for archiving or purging obsolete or redundant data. This helps optimize storage resources, improve data freshness, and mitigate risks associated with funnelling outdated or irrelevant data into AI models.

Reimagine monitoring and observability with Secoda

‍
We built Secoda to connect data quality, observability, and discovery into one streamlined platform. Our solution consolidates their data catalog, quality, and documentation tools into one place to help data teams reduce their data sprawl and streamline their infrastructure as well as costs. Secoda includes several features that help surface potential data quality issues in the same place where data is being explored:

• Data cataloging powered by AI, allowing teams to easily access all company data and align on metrics. It enables quick access to all data and helps users find answers to common questions within seconds.
• Data monitoring that helps keep eyes on data and optimize the health of pipelines, processes, and data infrastructure as a whole.
• Automated data lineage that improves data quality by tracking data from its source to its destination, making it easier to identify and address errors as well as relationships between dashboards and sources.

Clean Data By Working Smarter, Not Harder

Placing the entire burden of cleaning data upon data teams is not feasible, especially as organizations continue to expand and ingest exponentially larger volumes of data. Adopting a proactive approach by involving data producers and implementing quality assurance at the initial data collection stages ensures that data is AI-ready from the start. 

To get data “clean enough,” data teams must leverage a tool that empowers data producers by consolidating and automating needs around observability, metrics alignment, and — most importantly — data access. That way, data cleansing goals can transform from a cost-intensive, headache-inducing pipedream into an achievable reality.

Secoda is the premier solution providing all-in-one data search, catalog, lineage, monitoring, and governance capabilities that simplify too-complex tech stacks. Get to know what the hype is all about. Check out a demo today.