Many data teams rely on Snowflake to store and analyze critical data, but tracking data movement from ingestion to application can be challenging. At Secoda, we’re committed to improving data visibility and governance by integrating deeper with the tools teams use every day. Our latest Snowflake integration enhancements provide real-time visibility into Snowpipe ingestion pipelines and greater transparency into Streamlit applications, helping teams confidently govern their Snowflake ecosystem.

Bringing Streamlit applications into Secoda

Streamlit, an open-source app framework for building interactive data applications, is now supported through Secoda’s Snowflake integration. This update allows teams to bring Streamlit applications into their data catalog, making it easier to trace data sources and understand downstream dependencies. 

• Discover and manage applications: Streamlit apps are now searchable and filterable within Secoda, allowing teams to quickly locate and organize them alongside other Snowflake assets.
• Track data lineage with ease: Secoda automatically maps the upstream tables powering each Streamlit app, providing visibility into data origins, transformations, and dependencies—all within the broader data catalog.
• Seamless access to applications: Direct URL links allow users to open Streamlit apps from within Secoda, creating a smooth handoff between discovery and analysis.
• Boost data governance and trust: Having access to dependencies and lineage in context helps teams streamline audits, build trust in data, and reduce troubleshooting time.

Understanding Snowpipe in Secoda

Snowpipe is Snowflake’s continuous data ingestion service, allowing data to be loaded automatically from external cloud storage like AWS S3. With new support in Secoda’s Snowflake integration, teams can now gain visibility into their Snowpipe processes, ensuring ingested data is fresh, traceable, and aligned with governance best practices.

• Monitor ingestion pipelines in real time: Snowpipes are now visible within Secoda, allowing teams to track how and when data flows into Snowflake without manual intervention.
• Trace data sources and transformations: Users can see exactly where data originates (e.g., S3), how it moves through Snowpipe, and where it lands in Snowflake tables—eliminating blind spots in data movement.
• Map dependencies with full lineage tracking: Secoda automatically connects Snowpipe processes to downstream datasets, providing a clear view of dependencies for better governance and troubleshooting.
• Ensure real-time data freshness: With built-in freshness monitoring, Secoda automatically detects when new data is ingested via Snowpipe, helping teams stay confident that they are working with the most up-to-date datasets.

By integrating Snowpipe insights into Secoda, teams can reduce manual effort, improve governance, and gain confidence in their data ingestion workflows—all within a centralized platform.

How it works

Once the necessary permissions are set up in Snowflake and the Secoda-Snowflake integration is refreshed, all relevant Streamlit applications and Snowpipe metadata are pulled into Secoda automatically. Users can explore these new assets using the same familiar interface they use for other Snowflake resources.

Step 1: Setting up permissions

Ensure that the correct permissions are granted in Snowflake to allow Secoda to retrieve Streamlit and Snowpipe metadata. This includes granting read access to the relevant schemas and objects.

Step 2: Refreshing the integration

Navigate to your Snowflake integration settings in Secoda and initiate a refresh. This pulls in the latest data, including any newly created Streamlit apps and Snowpipe configurations.

Step 3: Exploring Streamlit dashboards and Snowpipes

Once the integration is updated, users can:

• Filter Streamlit apps in the Secoda catalog to locate dashboards.
• Trace data lineage to understand which tables and Snowpipe configurations contribute to a given Streamlit dashboard.
• Access Streamlit dashboards via direct links for quick navigation.

Example use case: Empowering data scientists with custom Streamlit dashboards

Data scientists often need custom-built dashboards to visualize and interact with datasets efficiently. Streamlit makes it easy to build such dashboards within Snowflake, offering an accessible way to analyze real-time data and share insights across teams.

Imagine a data science team has built a Streamlit-based orders dashboard in Snowflake. The dashboard pulls data from multiple tables, including customer information that is ingested using Snowpipe from an S3 bucket. With this new integration:

• Secoda automatically recognizes the Streamlit orders dashboard and shows it as a distinct asset in the catalog.
• Data scientists can trace exactly which Snowflake tables contribute to the dashboard, ensuring that models and visualizations are based on the correct datasets.
• By clicking on the lineage graph, teams can follow the entire data journey from ingestion through analysis, ensuring data governance and accuracy.
• Collaboration becomes easier as other team members can quickly access and understand how data flows into the dashboards they rely on for decision-making.

With these capabilities, data scientists can focus on building insights and improving models, rather than manually tracking down where their data is coming from.

What’s next

This release marks another step in our mission to provide end-to-end visibility into data ecosystems. By integrating real-time Snowpipe monitoring and ensuring full transparency of Streamlit applications, we’re making it easier for teams to manage, track, and govern their Snowflake environments with minimal manual effort.

Try it now: Refresh your Snowflake connection in Secoda to start exploring these new capabilities today. For a step-by-step guide on setup and permissions, check out our documentation.