How Transformco improved data discovery and cataloging across 400+ databases

Overview

Transformco is one of the largest integrated retailers in the US, focused on seamlessly connecting digital and physical shopping experiences to serve its members. In 2019, Transformco acquired Sears Holdings Corporation, including iconic brands such as Sears, Kmart, and Shop Your Way, along with subsidiaries like Sears Home Services. Following this acquisition, Transformco faced a significant challenge: a mountain of outdated data systems. The legacy infrastructure, spanning hundreds of databases and data silos, was holding back progress. To succeed in the digital-first retail environment, Transformco needed to modernize its data systems, ensure governance, and make data accessible across the company. At the start of 2024, Transformco underwent a significant digital transformation via our mainframe to cloud migration.

Enter Secoda—a platform that would help them streamline this complex transition, consolidate their scattered data, and bring clarity to their data. With Secoda, the data science team at Transformco embarked on a transformation journey, focused on reducing redundancy, enhancing governance, and future-proofing their data environment for long-term success.

The results

Through its partnership with Secoda, Transformco was able to completely transform their data management processes:

• Improved data discovery by merging over 400 databases into a single source of truth using Secoda’s catalog, reducing redundant data and enabling faster, more accurate decision-making across teams.
• Modernized their data infrastructure by migrating from outdated mainframe systems to Google Cloud Platform (GCP) and Snowflake, resulting in more agile and streamlined data governance across the company.
• Improved data quality standards using Secoda’s Data Quality Score (DQS), ensuring more reliable insights and reporting.
• Cut query-building time in half with Secoda’s API and Generative AI, dramatically speeding up the process of generating optimized Snowflake queries and freeing up the data team for more strategic tasks.

Setting clear goals for transformation

From the beginning, Transformco’s mission was clear:

1. Consolidate legacy data: Bringing order to a fragmented data landscape by consolidating over 400 databases, defining ownership, and modernizing legacy data systems with Secoda’s platform.
2. Streamline documentation: By ensuring consistent documentation and eliminating knowledge silos, Transformco sought to make data accessible and understandable to both technical and non-technical teams.
3. Enhance data quality: With the shift to modern tools like dbt and Snowflake, the team aimed to improve data quality processes while documenting workflows.
4. Implement AI-driven automation: Automating tasks like data descriptions using Secoda’s AI capabilities, making the transition from legacy systems more efficient.

The challenge

The road to data modernization was far from simple. Transformco inherited decades of legacy data—scattered across disconnected systems, many of which were poorly documented or completely orphaned. Key challenges included: 

1. Lack of data ownership: Many databases were still in use but lacked clear ownership. Teams struggled to identify who was responsible for managing or maintaining these data sources.
2. Orphaned and undocumented systems: Large portions of the data stack were orphaned, meaning no one actively managed or even knew the contents of many tables and databases. This created inefficiencies and prevented the team from optimizing their data governance.
3. Fragmented knowledge: Much of the institutional knowledge about the data stacks was tribal, residing only in the minds of key team members without being properly documented.
4. Difficult integration with modern tools: Without proper documentation, moving from legacy systems to modern platforms like dbt and Snowflake was an uphill battle.

How Secoda helped Transformco turn the corner

Secoda proved to be a game-changer in addressing these challenges. Read on to see how the platform made an impact.

A roadmap for modernization

Secoda allowed Transformco to consolidate over 400 databases, helping the team document their legacy systems and eliminate orphaned processes. By centralizing these previously disconnected data sources, Transformco could assign clear ownership to key data sets, which not only improved accountability but also reduced inefficiencies caused by redundant data. This clear ownership ensured that teams understood who was responsible for managing each data source, improving governance and streamlining data management.

"What Secoda is allowing us to do is look at where we’ve got multiple parts of the company creating similar data sets, and through the data science team, we are creating our Data Science Mart as the single source of truth for our analytics and marketing efforts,” says Melissa Blount, a key leader in this transformation.

As part of this modernization, Secoda played a key role in ensuring a seamless transition to modern platforms like dbt and Snowflake, aligning legacy and modern workflows. This smooth integration helped prevent disruptions and ensured data governance efforts remained intact throughout the migration.

Streamlining the data landscape

Before Secoda, vital information was scattered across teams, trapped in people’s heads. The platform helped Transformco build a central repository of knowledge, reducing dependence on individual team members and making it easier for new hires to get up to speed.

​​Thanks to the proactive efforts of the Data Science team, Secoda’s built-in templates were created to ensure consistency in their documentation efforts. Custom tags were implemented to certify which assets were production-ready and track each asset's development timeline.

“We wanted to keep source definitions aligned across databases and platforms,” Melissa said. “Using custom tags allowed us to easily track development timelines, making it easier for teams to know what’s ready for production.”

A journey toward automation and efficiency‍

Transformco leveraged Secoda’s AI-powered chatbot to automate the creation of data descriptions, significantly reducing manual workloads. The Data Science team’s strategic customizations ensured that these AI-generated descriptions met their unique standards without requiring manual review. This automation allowed Transformco to manage and document vast amounts of legacy data with ease, freeing up valuable time for more strategic and high-impact tasks.

Key customizations included:

• Starting descriptions with the tag "AI-Gen" for clear identification.
• Using brief, concise descriptions without full sentences to ensure clarity.
• Avoiding references to the associated database, column, or table to maintain a generalized approach.
• Expanding abbreviations in column names to ensure uniformity across datasets.
• Avoiding editorializing or the use of verbs like "Represents" or "Contains" to keep descriptions purely factual.
• Propagating definitions to downstream tables and views for consistency.

But automation didn’t stop at documentation. By utilizing Secoda’s API and Generative AI tools, Transformco was able to optimize Snowflake queries. By querying table and column descriptions directly from Secoda’s catalog, the team gained deeper insights into their data structure, enabling them to build more efficient queries. This not only streamlined workflows but also improved data accessibility across teams, allowing them to act more quickly on key business insights.

Improving data quality

Data quality was another critical pillar of Transformco’s transformation journey. Using Secoda’s Data Quality Score (DQS) as part of their checklist workflow, the team could verify the quality of each data set before it was moved into production. This real-time assessment ensured that only high-quality, verified data made it to production, reducing errors and increasing trust across departments.

“We’ve started using the Data Quality Score as part of our checklist before releasing something into production,” Melissa explained. “It helps us see what’s missing and identify which monitors we need to put in place.”

Looking ahead: What’s next for Transformco?

Transformco’s automation journey is ongoing. While the team has made impressive strides in consolidating databases, streamlining documentation, and improving data quality, the Data Science team’s leadership continues to be central to the transformation. With plans to roll out Secoda’s tools to more portfolio companies, they are focused on building a fully governed, accessible, and efficient data environment that will support their future growth.

"We are grateful for the opportunity to partner with Secoda as our vendor platform and have enjoyed working with their leadership team. In particular, I'd like to extend our appreciation to Etai Mizrahi, Secoda's CEO, for his outstanding leadership and commitment to our partnership. Etai and his team have consistently demonstrated responsiveness and availability, ensuring that our needs are met promptly and effectively. Their collaborative approach and ability to adapt to our requirements have made Secoda an invaluable partner in driving our goals forward. We look forward to continuing this productive relationship and achieving even greater success together." – Kapil Kanagal, Chief Data Scientist.

All of these advancements have significantly enhanced data discovery, making it easier for teams to quickly locate, understand, and utilize key data assets across the organization. By centralizing data and improving cataloging efforts, Transformco has empowered its teams with better, faster access to the insights they need.

By continuing to leverage Secoda’s AI-driven tools and its powerful API capabilities, Transformco is on track to fully modernize its data governance and documentation processes, ensuring long-term success and innovation.

‍