How Panasonic manages data at scale

Crafting and executing a data strategy within large organizations is challenging due to the sheer volume of data, organizational complexities, and the intricacies of people management. All of these factors make it difficult to make sense of and use data efficiently. Read on to learn how the Analytics Engineering team within The Smart Mobility Office manages scale and complexity as they support business units across Panasonic North America.

The situation

The Smart Mobility Office is the innovation unit within Panasonic responsible for developing products to improve transportation safety, energy efficiency, connected technologies, and related infrastructure.

As a centralized data team which supports the North American business units, Kevin Bennett (Data Engineering Manager), Matt Freyre (Analytics Engineer), and the rest of the Smart Mobility Office are tasked with managing an increasingly complex data infrastructure. In terms of scale, they process upwards of 3TB of connected vehicle, connected intersections, telematics, IoT, weather, and geospatial data each week to be shared with data consumers.

“At the core of all that is Secoda. It’s a one-stop shop for all documentation and anything we want to share across the business. Secoda connects all of our infrastructure together.”

‍

The challenge

Panasonic needed a solution to streamline data discovery and accessibility due to the vast volumes of generated and consumed data. Lack of a clear source for their company data posed challenges in facilitating understanding and usability.

Comprehensive documentation for data context, structure, and purpose was identified as a priority. The chosen solution had to be intuitive for both data practitioners and business stakeholders, establishing a solid foundation for the company's trust in data.

The team faced a critical challenge in leveraging data for shaping business direction through empowering users with essential metadata such as how certain fields are calculated, the upstream and downstream dependencies that feed into their data models, refresh schedules, any summary statistics, entity relationships, etc. The challenge was to transparently establish the foundation, enabling faster product deployment, efficient data handoff, and a tight feedback loop for continuous improvement.

‍

The solution

To effectively manage their substantial volume of data, Panasonic constructed their modern data stack using Secoda, Snowflake, Prefect, Soda, and dbt.

Secoda functions as the primary hub for their centralized data team by serving as a single source of truth to unlock data discovery. 

“It's a great tool that makes it easy for both data and business users to navigate and use data. It’s very intuitive and we don’t have to hand-hold users through the platform. This is a key component of our core initiatives to build the trust of our data consumers.”

On the data consumer side, Panasonic leverages Secoda’s discovery features to enable efficient searching and exploration of available datasets which creates a more agile and informed decision-making process. Ultimately, tools like Secoda bridge the gap between data creators and users, fostering a more collaborative and efficient data ecosystem.

"We use Secoda to provide documentation behind the logic, calculations, and papertrail behind the dashboards they use so they can trust it.”

To align efficient data access with effective data governance practices, their approach involved using Secoda’s role based access controls (RBAC) to ensure that data consumers only see the data that has been provisioned for their specific role. The analytics engineering team established schemas for each business unit which dictates the direction for how they ingest data. This structured approach fosters trust in curated data, ensuring users only encounter pertinent information, preventing distraction or confusion. Additionally, it also prevents unnecessary data ingestion. 

‍

Additional strategies to unlock data discovery

Four examples of Panasonic’s approach to unlocking data discovery: 

1. Implemented a dedicated business-unit schema which maps directly to their Secoda implementation of RBAC and documentation structure of their tables. Secoda enables self-serve analytics by creating a layer of definitions and data sources for business users to access.
2. Leveraged the Questions feature in Secoda as a forum to consolidate and serve answers to frequently asked questions that business stakeholders can access. By having this system in place, the analytics engineering team is never answering the same question twice and it provides a repository for data consumers to search for answers to their questions before submitting new ones.
3. Conduct regular office hours using Secoda to promote the feedback loop across all of their processes. All questions are documented in Secoda for easy reference in the future.
4. Based on frequently asked questions and requests, the team creates canned queries that can be shared across all departments through Secoda as their central repository for accessing data.

Learn more about Secoda

Secoda offers a comprehensive platform that streamlines data management, monitoring, governance, and discovery. By leveraging Secoda, enterprises can overcome scalability challenges, reduce time-to-insight, and unleash the full potential of their data assets.

Get a tour of the product and get started with an account here.

‍

‍