How TextUs used Secoda to reduce the volume of inbound data requests by over 50%

The challenge

TextUs is the industry leading consumer engagement platform with a mission to help businesses understand and engage with their customers through application to mobile two-way text messaging.

With the accelerated growth that TextUs has experienced, the volume of data required for critical business reporting and to generate insights quickly grew beyond what their internal data reporting pipeline could handle. As a data driven company, the internal reporting pipeline was unable to scale and became a blocker for meeting the expanding data needs of the company as a whole. 

Julie Garst, TextUs’s Data Science Team Lead, and the rest of the Data and Analytics team took on the challenge of building a modern stack that would scale with the company’s increased demand for data enablement. Using Secoda, Julie was able to successfully roll out a full data pipeline rebuild while automating the documentation process along the way. This resulted in a reduction of data requests by over 50% and resulted in a full company wide adoption of new dashboards and self-serve analytics tools to power go-to-market decisions.

"Having things documented on an ongoing basis has reduced the volume of requests the data team receives by over 50%"
_{Julie Garst, Team Lead}

The solution

Phase 1: Rebuild the internal data reporting pipeline to handle for scalability as the company continues to grow

Phase 2: Migrate data visualization platforms to enable business stakeholders with data in a faster, more efficient way than before

Phase 3: Onboard business stakeholders and non-data team members to the new platform with the least amount of friction to optimize adoption of their data products

Phase 1: A modern data stack approach to scalability 

"We needed to migrate our internal data pipeline to a modern stack to accommodate the level of growth we were experiencing as a company. The way you call data at several million rows is completely different from how you would handle billions of rows."

Julie and her team needed to move from their existing data stack towards one that could scale with the data needs of the business. The first step in replacing the old stack with a more sophisticated one was to understand how data was currently being aggregated from the application database to the warehouse, how it was being transformed, and verify the reporting schema that was feeding into their visualization platform for business users to consume. Without adequate documentation, it was extremely difficult to know how to move forward with the rebuild and have the confidence to safely manage change. To solve this issue, Julie used Secoda’s ability to automatically generate documentation and visualize the lineage between each column and table. Generating documentation and lineage helped Julie’s team understand all dependencies that were intertwined in their existing data pipeline, and allowed them to confidently ship changes without risking any critical data outages, shaping the direction of what needed to be done for the rebuild. 

The next consideration was how to build their new stack while efficiently making the knowledge behind the changes easily accessible. This allowed the TextUs team to not only scale the infrastructure, but also scale the ability to enable consistent and repeatable data governance protocols. As the data engineering team built a more efficient reporting schema in Snowflake, Secoda would capture their comments relating to the newly generated tables and automatically create documentation and lineage graphs for them. This streamlined how Julie could scale the team’s documentation ability and save them countless hours of manual effort.

Secoda allowed them to

• Eliminate the need to investigate what new tables and columns were created and where they were located
• Save engineering resources by not needing a data engineer to translate all comments to documentation
• Automatically create new documentation for the new tables and infrastructure
• Set themselves up for success as they moved into the next phase of the migration: extracting the raw data from their warehouse and putting it into a new reporting schema that runs quickly

"Everything was automatically documented for us as we rebuilt our new data pipeline. Now, if something breaks or when we need to rebuild something and we need to figure out where the data is coming from, it's already documented for us. Documentation is always the first thing that gets dropped when you're building something new so it’s one less thing to dedicate resources to."

Phase 2: Upgrade reporting schema and replace visualization platform

Replacing a visualization platform is a massive undertaking. Julie and her team migrated from Looker to Sigma and used Secoda to solve the key challenge of how to teach business stakeholders to use a brand new reporting suite, while building and designing the technical elements at the same time.

Because of the custom nature of LookML, creating documentation to verify the integrity of data that feeds into the dashboards was extremely difficult on Looker. It was imperative to understand how the data was queried given the vast downstream impact of how that data is displayed to the end consumer. Combined with the move from Looker to Sigma, Secoda's automated data verification process has allowed Julie and her team to quickly and accurately verify the data sources of their business dashboards without having to spend hours manually checking each data source. This has allowed them to make more informed decisions and to better understand the data they are working with.

"Before Secoda, in order for us to figure out how data was being pulled into Looker (so we could rebuild it in Sigma), we would have had to go to Looker, and analyze the LookML model code behind it to see how it was joining data. We would then need to figure out which table it was coming from in Snowflake. From Snowflake, we would have to go to dbt to see how it was grabbing that data from somewhere else in Snowflake, and then from that Snowflake DB we had to checkout both Airbyte and our product database to complete the whole chain to figure out where the data is coming from. It's like 20 steps to verify your data. Whereas with Secoda, we could move along quickly."

Phase 3: Flawlessly onboard business stakeholders to increase adoption of new data products

On the business adoption side of the challenge, Julie needed to find a way to not only train colleagues outside of the data team on how to use Sigma, but also how to find old reports they were familiar with in this new platform.

Julie and her team created a homepage for their data by requiring all business users to access user guides and FAQs in Secoda as their source of truth before navigating to the dashboard they were searching for. By sending the business users to Secoda first, all relevant reports were consolidated in one place where they could simply use natural language search to find the report they needed. The Data and Analytics team created documentation for every new dashboard that was rebuilt within this new pipeline. Using Secoda as the first access point for business users to view and use dashboards is a workflow Julie credits as a reason why launching a new company-wide dashboarding tool happened so flawlessly:

"The people at the company who are most excited are on the non-technical side. They like that they can go in and see and understand why we're calculating something in a certain way, or why we're asking for something with particular terminology. It makes them feel able to actually interpret the data that we give them. They don't have to always say, “well, let me ask my data person”. They can pull up the dashboard and be self-serve."

Conclusion

Julie and her team developed an invaluable, company-wide sense of trust in their data and leaned on Secoda to facilitate this roll-out. It has enabled them to ensure that the data they are working with is accurate and up-to-date, while delighting business teams with access to the data they need without relying on the data analysts. Julie used Secoda to rebuild the  data reporting pipeline and automated documentation. Data requests decreased by over 50%. The whole company now uses new dashboards and self-serve analytics tools to make go-to-market decisions.