How Vanta seamlessly migrated to a modern data warehouse with Secoda

Learn how Vanta successfully migrated to a modern data warehouse with Secoda, improving data discoverability, lineage, and documentation. The key results: faster reporting, zero downtime, and automated data management processes.

Data management and migration present significant challenges for companies aiming to maintain efficiency and scalability.

Vanta is the leading trust management platform and is one of the most innovative companies in the space. Vanta’s mission is to secure the internet and protect consumer data - and to uphold this mission, their data team holds themselves to a very high bar.

The team faced the challenge of modernizing its legacy data warehouse head-on, migrating to Snowflake. This case study explores how Secoda helped facilitate this critical migration and created healthy data management practices - from centralizing documentation to understanding how data was used across the organization.

The results

• 4-month migration to Snowflake
• Zero downtime across data assets
• 80% reduction in report latency 
• Moving from 0 to 1 in data discoverability and lineage 
• A completely automated documentation process 

The goal

‍Modernize Vanta’s data warehouse without disrupting operations, and build healthy data management practices around documentation, lineage, and data literacy at the company.

The challenge

‍The Vanta team faced a complex task: migrating their data warehouse to Snowflake. Existing data environments lacked comprehensive documentation, complicating the migration process.

Automating security compliance for businesses 

Vanta was founded in 2018 in response to an increasing number of high-profile data breaches, to address the critical need for enhanced online security. With online security becoming more and more important, Vanta’s mission is to protect consumer data and restore trust in internet business. The team understands firsthand how hard it can be for fast-growing companies to invest the time and people resources it takes to build a solid security foundation. Vanta has grown to be one of the most innovative companies in the security compliance space. Any operation that deals in secure, sensitive information must develop strategies on how to manage this data, and Vanta is no exception. Considering the nature of their business, they also hold themselves to a very high standard.

“Data is the lifeblood of the business,” said Jake Peterson, Head of Data at Vanta. “We need that direction to know what we’re doing, and what we’re doing well.”

Vanta’s data team manages huge amounts of data coming in across 30+ SaaS applications for various teams, on top of their proprietary product data. The team uses data from the product to optimize the customer journey, help customers to develop their security stacks and inform product decisions. “Our biggest principle for the data team while managing our proprietary data is to maintain the trust of our customers. We hold ourselves and the data and analytics team to an extremely high bar,” said Peterson. 

The need to upgrade to a modern data stack

Vanta’s rapid growth necessitated an upgrade to a modern data stack, prioritizing both speed and efficiency. This upgrade included adopting a data warehouse capable of handling increased demand and a discovery tool to optimize data management processes across the organization.

To support their upcoming migration to Snowflake with minimal operational disruption, Vanta selected Secoda—an AI-powered platform for data search, cataloging, lineage, and documentation. Secoda provided the comprehensive visibility required to manage information seamlessly as it moved through Vanta’s systems, ensuring data was effectively tracked and utilized across various teams.

Managing Vanta’s product data, which flows through MongoDB in a schema-less format before being structured in the data warehouse, presented challenges due to the lack of centralized documentation. With Secoda, Vanta was able to automate documentation, streamline data management, and enhance clarity for both current team members and newcomers, resulting in a smoother migration and more efficient operations.

How Secoda assisted in the migration process

Secoda offers a suite of tools that address Vanta's needs for seamless migration and robust data management.

• Efficient migration: Leveraging Secoda along with other integrated data tools, Vanta successfully migrated to Snowflake within four months, with zero downtime and significant performance improvements. Pre-migration query times were reduced dramatically from minutes to under 30 seconds, greatly enhancing user satisfaction and engagement.

• Leveraging Secoda’s API: Vanta leveraged Secoda’s API to migrate pre-existing documentation into Snowflake so that work wouldn’t have to be lost. “We didn’t have to throw away the documentation that already existed. We used Secoda’s API to repurpose that documentation and bring it over to Snowflake with Secoda, which was a huge time saver for us,” said Barber.

• Migrating dashboards: Near the end of the migration process, the team needed to migrate their Sigma dashboards, but faced the challenge of not understanding what dashboards were being actively used.

‍“Secoda being able to find dashboards, see how often they are being queried, and how much they are being used has helped us save time on painful migration work. We ended up only migrating 125 dashboards, and realizing we didn’t need the others. We wouldn’t have been able to do that without Secoda.”

Using Secoda the team was able to identify which dashboards could be deprecated, saving them time and effort.

• Enhanced data management: Secoda's platform provides a central repository for Vanta's data documentation, significantly improving data hygiene and operational efficiency. It allows for the automation of documentation processes, ensuring that data changes are accurately reflected in real-time across the organization.

• Automated documentation: The easy integration between Secoda and Vanta’s data pipelines allows for automatic updates to data documentation, reducing manual effort and ensuring accuracy. This automation extends to managing data contracts and sensitive data, further streamlining Vanta's data governance practices. 

“We now have a CI/CD script that extracts from the dbt manifest and pushes it into Secoda, Secoda is auto-documenting these tables, providing links to contract documentation, all automatically. It's changing the game for us."

Post-migration, Vanta experienced a transformative improvement in its data operations. “The migration process was a huge success in the company, and was extremely painless,” said Peterson.  

Query performance saw a dramatic increase, operational efficiencies were realized through better data management practices, and the data team could focus on higher-value tasks, thanks to automation. 

"I'm gonna brag for a second because I think this is the best project I've done in my career,” said Barber. “We did it, we launched it on time, we had no downtime... This was a huge improvement, and a big win for our stakeholders.” Barber described the ins and outs of this migration and recognized that it is a process that rarely goes perfectly. 

“Migrating 100% of everything perfectly is going to be very difficult and expensive,” he shared. “It’s best to go in and make a goal to grab the most important 80-90% of what you need. Secoda was paramount in helping us understand what was important - using the popularity metrics helped us know what data was being used frequently, and what to focus on.”

Looking ahead: How Vanta uses Secoda to maintain good data hygiene 

With the foundation set with Secoda, Vanta plans to further explore data governance features, such as sensitive data management, usage monitoring and data contracts. The successful migration and improved data practices have positioned Vanta to scale its operations efficiently.

Vanta's journey with Secoda underscores the critical importance of robust data management and efficient migration strategies in today's data-driven world. By leveraging Secoda, Vanta not only navigated a complex migration but also established a framework for sustainable data governance and documentation practices for the future.