NestJS Kafka Client

A production-ready NestJS module for Kafka client and consumer functionality built on top of kafkajs. This library provides enterprise-grade features including intelligent batch processing, idempotency guarantees, key-based grouping, and automatic pressure management.

Quick Start

npm install @jescrich/nestjs-kafka-client

import { KafkaModule, ConsumerModule } from '@jescrich/nestjs-kafka-client';

@Module({
  imports: [
    KafkaModule.forRoot({
      clientId: 'my-app',
      brokers: ['localhost:9092'],
    }),
    ConsumerModule,
  ],
})
export class AppModule {}

Documentation Sections

• Introduction & Technical Deep Dive
• Installation & Setup
• Kafka Producer
• Kafka Consumer
• Advanced Features
• Configuration
• Best Practices
• Troubleshooting

Key Features

Core Functionality

• Kafka Client: High-performance Kafka producer with intelligent connection management
• Kafka Consumer: Enterprise-grade consumer with advanced batch processing capabilities
• Built on KafkaJS: Leverages the robust kafkajs library with additional enterprise features

Advanced Processing

• Intelligent Batch Processing: Automatically groups messages for optimal throughput
• Key-Based Grouping: Groups messages by key within batches for ordered processing
• Idempotency Support: Built-in mechanisms to prevent duplicate message processing
• Back Pressure Handling: Automatic throttling when downstream systems are overwhelmed
• Front Pressure Management: Smart buffering and flow control for incoming messages

Reliability & Monitoring

• Dead Letter Queue (DLQ): Automatic handling and routing of failed messages
• Health Monitoring: Comprehensive health checks for Kafka connections and consumers
• Connection Management: Automatic connection pooling, reconnection, and failover
• Graceful Shutdown: Proper cleanup and message completion on application shutdown

Performance Benefits

• Up to 10x throughput improvement with intelligent batch processing
• Maintains message ordering while maximizing parallelism through key grouping
• Automatic pressure management prevents system overload
• Memory efficient streaming processing for large message volumes