Job Overview: We are seeking an experienced Kafka Streaming Platform Engineer to design, develop, and manage real-time data streaming solutions using Apache Kafka. This role is essential in building and optimizing data pipelines, ensuring system scalability, and integrating Kafka with various systems. The engineer will play a key role in maintaining and optimizing Kafka clusters, working with Kafka Connect, Kafka Streams, and other associated technologies. You will collaborate with cross-functional teams, including data architects, DBAs, and application developers, and leverage your experience with DevOps practices (CI/CD, Docker, Kubernetes) for automated deployments and infrastructure management. Strong problem-solving skills and a solid understanding of event-driven architectures are essential for this role. Key Responsibilities: 1. Data Pipeline Development Design, develop, and maintain data pipelines using Apache Kafka, including real-time streaming applications. Build highly scalable data solutions that can handle large volumes of data in real-time. 2. Kafka Cluster Management Install, configure, monitor, and optimize Kafka clusters, ensuring high availability and efficient resource utilization. Manage Kafka Connect and ensure smooth integration with external systems (databases, messaging systems, etc.). 3. Event-Driven Architecture Implementation Design and implement event-driven architectures using Kafka Streams, Kafka Connect, KSQL, and other Kafka-related technologies. Leverage pub/sub patterns to enable real-time event processing and improve system responsiveness. 4. Integration with Other Systems Integrate Kafka with various databases (relational, NoSQL), applications, and other data processing platforms. Utilize Change Data Capture (CDC) techniques using Kafka Connect, Debezium, or custom connectors to enable real-time data sync across platforms. 5. Performance Optimization Ensure data ingestion and transformation pipelines are optimized for performance, reliability, and scalability. Continuously monitor and improve Kafka cluster performance, minimizing latency and maximizing throughput. 6. Troubleshooting and Monitoring Proactively monitor Kafka clusters to identify and resolve any performance or availability issues. Troubleshoot complex data streaming issues and ensure high uptime and system stability. 7. Code Quality and Best Practices Apply software engineering best practices such as object-oriented programming (OOP), TDD, and design patterns to ensure code maintainability and quality. Write efficient, clean, and well-documented code to meet project requirements. 8. DevOps Practices Utilize DevOps methodologies (CI/CD, Docker, Kubernetes) for automated deployments and infrastructure management. Automate infrastructure provisioning, Kafka cluster scaling, and deployment workflows. 9. Documentation Create and maintain technical documentation, including data flow diagrams, design documents, and operational procedures. Ensure that all processes and architecture are well-documented and easily understood by the team. 10. Collaboration and Stakeholder Communication Collaborate with cross-functional teams, including data architects, DBAs, and application developers, to align on project goals and technical solutions. Provide technical support and guidance to team members, ensuring high-quality implementations. Requirements Required Skills and Experience: 8+ years of experience with Apache Kafka, including expertise in Kafka Connect, Kafka Streams, and Kafka brokers. Strong proficiency in Java, with experience in object-oriented programming (OOP). Shell scripting and Python experience (desirable). Solid understanding of event-driven architectures, pub/sub patterns, and real-time data processing. Database knowledge: Experience with MongoDB, relational databases, or other data storage solutions. Experience with cloud platforms such as AWS, Azure, or GCP. Solid experience with DevOps practices including CI/CD, Docker, and Kubernetes. Strong troubleshooting and problem-solving skills, with an ability to identify and resolve complex issues in Kafka clusters. Schema registry experience and familiarity with tools related to Kafka (e.g., Confluent, Schema Registry, Control Center) is a plus. Experience with Change Data Capture (CDC), Debezium, or similar tools is highly desirable.