# Kafka Configuration for Freeleaps Data Platform

This directory contains the Kafka configuration for the Freeleaps Data Platform.

## Files

- `values.yaml` - Original Bitnami Kafka Helm values (legacy)
- `vpa.yaml` - Vertical Pod Autoscaler configuration
- `strimzi-kafka-cluster.yaml` - Strimzi Kafka Operator cluster configuration
- `README.md` - This file

## Migration from Bitnami to Strimzi

### Current State
- **Existing**: Bitnami Kafka deployment (single node)
- **Target**: Strimzi Kafka Operator with HA cluster (3 brokers + 3 ZK nodes)

### Migration Steps

1. **Install Strimzi Operator** (already done)
   ```bash
   helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
     --namespace freeleaps-data-platform
   ```

2. **Delete existing Kafka deployment**
   ```bash
   kubectl delete statefulset kafka-controller -n freeleaps-data-platform
   kubectl delete service kafka kafka-controller-headless -n freeleaps-data-platform
   kubectl delete configmap kafka-controller-configuration -n freeleaps-data-platform
   kubectl delete pvc data-kafka-controller-0 -n freeleaps-data-platform
   ```

3. **Deploy new Strimzi Kafka cluster**
   ```bash
   kubectl apply -f strimzi-kafka-cluster.yaml
   ```

### New Cluster Features

- **High Availability**: 3 Kafka brokers + 3 Zookeeper nodes
- **Authentication**: SCRAM-SHA-512
- **Storage**: 20Gi per broker, 10Gi per ZK node
- **Replication**: 3x replication factor
- **Security**: TLS support for internal communication

### Monitoring

Check cluster status:
```bash
kubectl get kafka -n freeleaps-data-platform
kubectl get kafkatopic -n freeleaps-data-platform
kubectl get kafkauser -n freeleaps-data-platform
```

### Client Configuration

For applications connecting to the new Kafka cluster, use:
- **Bootstrap servers**: `kafka-cluster-kafka-bootstrap.freeleaps-data-platform.svc.freeleaps.cluster:9092`
- **Authentication**: SCRAM-SHA-512
- **Username**: `freeleaps-user`
- **Password**: Generated by Strimzi (check the secret)

## Storage Requirements

- **Kafka brokers**: 3 × 20Gi = 60Gi
- **Zookeeper**: 3 × 10Gi = 30Gi
- **Total**: 90Gi

## Network Policy

The cluster includes network policies that allow:
- Internal cluster communication
- External client access (with proper authentication)
- Metrics collection (if enabled)