Kafka Deduplicator
CLI Verified Active🦔 PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.
To provide reliable, partition-aware deduplication of Kafka events, ensuring data integrity and operational stability in streaming pipelines.
Features
- Partition-aware Kafka event deduplication
- State persistence using RocksDB
- Checkpointing for recovery and reassignment
- Support for multiple pipeline types (ingestion_events, clickhouse_events)
- Configurable fail-open mode for operational bypass
Use Cases
- Ensuring unique event processing from Kafka topics
- Recovering from consumer rebalances or failures with minimal data loss
- Handling high-throughput event streams with deduplication requirements
- Integrating with downstream Kafka topics or other data sinks after deduplication
Non-Goals
- Providing a generic Kafka consumer without deduplication logic
- Acting as a Kafka producer or broker
- Handling event transformations beyond deduplication
Trust
- info:Issues AttentionThere were 544 issues opened and 163 closed in the last 90 days, indicating a closure rate below 50% and a high volume of open issues.
Compliance
- info:GDPRThe service processes Kafka events which may contain personal data. While not submitting to a third party, the data is processed and potentially logged, with no explicit sanitization mentioned before LLM interaction.
Quality Score
VerifiedTrust Signals
Similar Extensions
Netdata Field Encoder CLI
99The fastest path to AI-powered full stack observability, even for lean teams.
Personhog Writer
75🦔 PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.
Personhog Leader
75🦔 PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.
Batch Import Worker
75🦔 PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.
Livestream
68🦔 PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.
Property Defs Rs
65🦔 PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.