Wed Sep 07 2022
Introducing Streaming Data Performance Monitoring: SDPM
by
Daniel Selans
What is Streaming Data Performance Monitoring (SDPM)?
SDPM is a sub-category of observability tooling that focuses on data within streaming systems.
The goal for SDPM is to help users identify and fix issues within their streaming architectures as quickly as possible before those issues impact downstream systems or business goals.
An SDPM solution achieves this by inspecting the data itself that passes through a streaming or messaging platform (such as Kafka or RabbitMQ). It usually consists of an agent that hooks into one or more data sources as a consumer and relays all data it comes across to a centralized data processing destination.
Think APM (application performance monitoring) but for data.
SDPM should be:
- Real-time
- SDK-less
- Hands-free
Don't Miss Our Latest Updates
What can SDPM do for you?
An effective SDPM strategy will enable you to:
- Improve developer productivity by reducing debug time.
- Improve your MTTR (mean time to recovery) during outages.
- Ensure your data adheres to your data quality standards.
- Provide early detection of unusual stream activity.
- Monitor data SLAs (service-level agreements).
- Comply with data privacy regulations (GDPR, CCPA, etc.).
Who is SDPM for?
SDPM is for anyone that makes significant use of streaming data architectures. This includes:
- Developers who build and maintain streaming applications.
- Data engineers who manage and monitor streaming data pipelines.
- Ops/site reliability engineers (SREs) who are responsible for the uptime and performance of streaming systems.
Why is SDPM important?
In the age of big data and real-time analytics, more and more businesses are relying on streaming architectures to process and route large volumes of data in near-real-time.
As these streaming data systems become increasingly complex, it becomes more difficult to identify and fix issues within these systems before it causes downstream problems.
This is where SDPM comes in.
SDPM tools help users detect issues within their streaming data systems by inspecting the data itself as it flows through the system. SDPM can detect and alert on data quality or schema changes, find anomalies in data sets and identify bottlenecks in your streaming architecture.
Use cases for SDPM
Some of the use-cases for SDPM include:
Data Observability
Provide engineers with the ability to inspect any data that has ever passed your streaming systems.
Data Validation
Ability to monitor and alert for data that does not adhere to your data standards.
For example: "Data must contain this field and it should be a valid `uuid`".
Data Governance
Ability to detect and alert when data includes unexpected PII or other sensitive data.
Disaster Recovery
Ability to repopulate your streaming systems with data as-they-were-before a catastrophic failure occurred.
Stream Lineage Visualization
Ability to visualize how data evolves and flows between systems.
Streaming Data Performance Monitoring FAQ
Daniel Selans
CTO
Dan is the co-founder and CTO of Streamdal. Dan is a tech industry veteran with 20+ years of experience working as a principal engineer at companies like New Relic, InVision, Digital Ocean and various data centers. He is passionate about distributed systems, software architecture and enabling observability in next-gen systems.
Continue Exploring
Wed Mar 01 2023
AWS SES Events: How We Use Streamdal For Event Monitoring
by
Fritz Stauffacher
Learn how we use our own Streamdal console for event monitoring of AWS SES events. A step-by-step guide.
Wed Feb 08 2023
Announcing Streamdal: A Streaming Data Performance Monitoring Product
by
Ustin Zarubin
Today we are excited to announce the next round of funding that we have secured and what is next for Streamdal.
Wed Oct 19 2022
What Is Protobuf And Why Should You Use It?
by
Ustin Zarubin
This serves as a nice primer on the two and gets you feeling like a data observability pro in no time.