Skip to main content

Feroze's musings on Technology

Search This Blog

Featured Post

Kafka Privacy Toolkit Part 1: Protect PII data using a custom single message transform.

July 07, 2024 cpra GLBA Kafka kafka-connect single-message-transform

Get link
Facebook
X
Pinterest
Email
Other Apps

Hi!

August 13, 2009

Get link
Facebook
X
Pinterest
Email
Other Apps

Welcome to my blog. Here, I will post my thoughts on Software, as well as code snippets and other stuff.

For my previous blog, you can see https://blogs.msdn.com/feroze_daud

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

Post a Comment

Popular Posts

KafkaAvroSerializer: Efficient way to serialize messages with Avro to a Kafka topic

Introduction Apache Kafka has become the platform of choice for doing large scale messaging. It supports a variety of serialization formats for messages, and also allows integration with external systems using a wide variety of connectors. Apache Kafka allows messages to be stored in a variety of formats in the topic: - Opaque bytes. - Avro - Json - Protocol Buffers In this post, we take a deep dive into the way the messages are stored in a Kafka topic, when the serialization format is Avro. Avro Serialization Apache Avro is an Open Source project. It is used to specify a schema for a record. It provides a schema definition language ( AVSC ). Avro allows a very space efficient serialization format, where the schema id of the field is stored along with the value. It allows for schema evolution as well. Avro schemas are created using JSON format. The format of the schema document is defined in RFC 4627 . An example Avro schema looks as follows: { "type": "reco...

Kafka Privacy Toolkit Part 1: Protect PII data using a custom single message transform.

Introduction Kafka is a highly scalable and durable distributed streaming platform that has become the go-to choice for handling large amounts of data across a variety of industries. Its ecosystem of services is vast and powerful, enabling businesses to solve complex problems and uncover insights from their data in real-time. One of the key services in the Kafka ecosystem is Kafka Connect. It is a framework that enables streaming integration between Kafka and other data systems. Kafka Connect is flexible and extensible, with a plugin-based architecture that allows it to integrate seamlessly with various data systems. Users can write custom connectors to integrate Kafka with any data system that has a Java API, and Kafka Connect's RESTful API can be used to configure and manage connector instances. Data Privacy & Compliance Organizations process an extensive amount of data and some of that data ...

mask-json-field transform - a Kafka Connect SMT to mask out PII data from JSON fields in Kafka messages.

Introduction In today's fast-paced digital era, where vast amounts of data are being generated and processed at an unprecedented scale, the need for efficient, scalable, and real-time data processing solutions has become crucial. Traditional messaging systems and data integration tools often struggle to meet these demands, leading to delays, data loss, and inefficiencies in data processing pipelines. That's where Apache Kafka, a powerful distributed streaming platform, comes into play. Kafka has emerged as a game-changer in the world of data streaming, enabling organizations to build robust, fault-tolerant, and highly scalable data pipelines. Developed by the Apache Software Foundation, Kafka was originally created to handle the real-time data ingestion and processing needs of LinkedIn. Since its inception, Kafka has gained immense popularity and has become the de facto standard for building event-driven architectures in a wide range of industries. Kafka Connect plays a pi...

Powered by Blogger