e7ec6a4258
Three Kafka nodes + Three ZooKeeper nodes Can tolerate 1 node failure 3 replicas distributed across different nodes Use Azure Disk SSD SCRAM-SHA-512 + ACLs
76 lines
2.3 KiB
Markdown
76 lines
2.3 KiB
Markdown
# Kafka Configuration for Freeleaps Data Platform
|
||
|
||
This directory contains the Kafka configuration for the Freeleaps Data Platform.
|
||
|
||
## Files
|
||
|
||
- `values.yaml` - Original Bitnami Kafka Helm values (legacy)
|
||
- `vpa.yaml` - Vertical Pod Autoscaler configuration
|
||
- `strimzi-kafka-cluster.yaml` - Strimzi Kafka Operator cluster configuration
|
||
- `README.md` - This file
|
||
|
||
## Migration from Bitnami to Strimzi
|
||
|
||
### Current State
|
||
- **Existing**: Bitnami Kafka deployment (single node)
|
||
- **Target**: Strimzi Kafka Operator with HA cluster (3 brokers + 3 ZK nodes)
|
||
|
||
### Migration Steps
|
||
|
||
1. **Install Strimzi Operator** (already done)
|
||
```bash
|
||
helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
|
||
--namespace freeleaps-data-platform
|
||
```
|
||
|
||
2. **Delete existing Kafka deployment**
|
||
```bash
|
||
kubectl delete statefulset kafka-controller -n freeleaps-data-platform
|
||
kubectl delete service kafka kafka-controller-headless -n freeleaps-data-platform
|
||
kubectl delete configmap kafka-controller-configuration -n freeleaps-data-platform
|
||
kubectl delete pvc data-kafka-controller-0 -n freeleaps-data-platform
|
||
```
|
||
|
||
3. **Deploy new Strimzi Kafka cluster**
|
||
```bash
|
||
kubectl apply -f strimzi-kafka-cluster.yaml
|
||
```
|
||
|
||
### New Cluster Features
|
||
|
||
- **High Availability**: 3 Kafka brokers + 3 Zookeeper nodes
|
||
- **Authentication**: SCRAM-SHA-512
|
||
- **Storage**: 20Gi per broker, 10Gi per ZK node
|
||
- **Replication**: 3x replication factor
|
||
- **Security**: TLS support for internal communication
|
||
|
||
### Monitoring
|
||
|
||
Check cluster status:
|
||
```bash
|
||
kubectl get kafka -n freeleaps-data-platform
|
||
kubectl get kafkatopic -n freeleaps-data-platform
|
||
kubectl get kafkauser -n freeleaps-data-platform
|
||
```
|
||
|
||
### Client Configuration
|
||
|
||
For applications connecting to the new Kafka cluster, use:
|
||
- **Bootstrap servers**: `kafka-cluster-kafka-bootstrap.freeleaps-data-platform.svc.freeleaps.cluster:9092`
|
||
- **Authentication**: SCRAM-SHA-512
|
||
- **Username**: `freeleaps-user`
|
||
- **Password**: Generated by Strimzi (check the secret)
|
||
|
||
## Storage Requirements
|
||
|
||
- **Kafka brokers**: 3 × 20Gi = 60Gi
|
||
- **Zookeeper**: 3 × 10Gi = 30Gi
|
||
- **Total**: 90Gi
|
||
|
||
## Network Policy
|
||
|
||
The cluster includes network policies that allow:
|
||
- Internal cluster communication
|
||
- External client access (with proper authentication)
|
||
- Metrics collection (if enabled)
|