Arsitektur Kubernetes Control Plane & Data Plane Skala Enterprise: Deep-Dive Konsensus Raft etcd v3, Kube-Apiserver APF, Gateway API v1.1+ Cilium L7, dan Linux Cgroups v2

=======================================================================================================
               KUBERNETES CONTROL PLANE & DATA PLANE INTERACTION ARCHITECTURE
=======================================================================================================
 [ Clients: kubectl / CI-CD / Controllers ] 
                    │ (Mutual TLS 1.3)
                    ▼
 ┌────────────────── kube-apiserver ─────────────────────────────────────────────────────────────────┐
 │                                                                                                   │
 │  Authentication ──► Authorization (RBAC/Node) ──► Mutating Webhooks ──► Validating Webhooks       │
 │                                                         │                                         │
 │  ┌──────────────────────────────────────────────────────┴──────────────────────────────────────┐  │
 │  │        API PRIORITY AND FAIRNESS (APF) DISPATCH ENGINE                                      │  │
 │  │  [ FlowSchema Evaluator ] ──► [ PriorityLevelConfiguration (Concurrency Seats) ]            │  │
 │  │                                              │                                              │  │
 │  │                                    ┌─────────┴─────────┐                                    │  │
 │  │                               [ Wait-Queue ]     [ Dispatch Seat ]                          │  │
 │  └──────────────────────────────────────┬─────────────────┬────────────────────────────────────┘  │
 └─────────────────────────────────────────┼─────────────────┼───────────────────────────────────────┘
                                           │                 │
                                           ▼ (gRPC Client)   ▼
 ┌─────────────────────────────────────────────────────────── etcd v3 (3-Node Raft Cluster) ─────────┐
 │                                                                                                   │
 │   [ Raft Consensus Engine ] ──(AppendEntries RPC)──► [ WAL Engine: synchronous fsync() to NVMe ]  │
 │              │                                                                                    │
 │              ▼                                                                                    │
 │   [ MVCC Storage Engine: bbolt B+ Tree ] ──(In-Memory B-Tree Index: key -> 64-bit Revision)       │
 └───────────────────────────────────────────────────────────────────────────────────────────────────┘
                                           │
                         Watch Notifications (gRPC HTTP/2 Streams)
                                           │
    ┌──────────────────────────────────────┴───────────────────────────────────────┐
    ▼                                                                              ▼
┌────────────────── Kubelet (Node Agent) ──────────┐  ┌────────────────── Cilium L7 Gateway Controller ──┐
│                                                  │  │                                                  │
│  [ PLEG (Pod Lifecycle Event Generator) ]        │  │  [ Gateway API Ingress Controller ]              │
│  [ CRI Client (gRPC) ] ──► containerd            │  │  [ eBPF Socket Layer / XDP Redirection ]         │
│  [ Cgroups v2 Manager: memory.high, memory.max ] │  │  [ Envoy Embedded Proxy Engine ]                 │
└──────────────────────────────────────────────────┘  └──────────────────────────────────────────────────┘

+---------------------------------------------------------------------------------------------------+
|                                 ETCD V3 DUAL-INDEX STORAGE ENGINE                                 |
+---------------------------------------------------------------------------------------------------+
|  [ In-Memory B-Tree Index ]                                                                       |
|  Key: "/registry/pods/prod/nginx"  ──► Value: Revision { main: 10452, sub: 1, create: 9100 }      |
|                                                                                                   |
|  [ Disk-Backed bbolt Engine (B+ Tree Database) ]                                                   |
|  Key: [Revision: 10452_1]          ──► Value: [ Protobuf Serialized Pod Data / Metadata ]         |
|  Key: [Revision: 10450_0]          ──► Value: [ Versi Lama Pod ]                                  |
|  Key: [Revision: 9100_0]           ──► Value: [ Versi Inisial Pod ]                               |
+---------------------------------------------------------------------------------------------------+

=======================================================================================================
                         API PRIORITY AND FAIRNESS (APF) SCHEDULING ENGINE
=======================================================================================================
 Incoming Request (HTTP/2)
          │
          ▼
 [ FlowSchema Classifier ] ────► Matches Subject & Resource Rules (e.g. system:nodes vs serviceaccounts)
          │
          ▼
 Assign to PriorityLevelConfiguration (e.g., 'workload-high', 'system', 'exempt')
          │
          ├── Is Exempt? ───────► Bypass Queuing & Concurrency Controls directly to Handlers
          │
          ▼
 [ Shuffle Sharding Engine ]
          │ Hash(Request Attributes) % HandSize
          ├── Assigns request to a virtual Queue among N Queues (Reduces Head-of-Line Blocking)
          │
          ▼
 [ Fair Queuing Scheduler ] (Modified Dual-Color Virtual Time Algorithm)
          │ Computes Virtual Finish Time (VFT)
          │
          ├── Queue Full? ──────► HTTP 429 "Too Many Requests" (Retry-After Header emitted)
          │
          ▼
 [ Concurrency Limit Execution Pool ] (Dynamic Concurrency Seats)
          │ Seat Available?
          ├── Yes ──────────────► Forwarded to etcd / Handler Pipeline
          └── No ───────────────► Queued until Seat Released or Request Context Timeout

+---------------------------------------------------------------------------------------------------+
| Infrastructure Provider (Platform Admin / Infra Team)                                            |
|   └── GatewayClass: Menentukan implementasi controller & tipe load balancer (e.g. Cilium / Envoy) |
+---------------------------------------------------------------------------------------------------+
                                  │ Defines Controller
                                  ▼
+---------------------------------------------------------------------------------------------------+
| Cluster Operator (DevOps / SRE Team)                                                              |
|   └── Gateway: Mengalokasikan IP publik, port (80/443), TLS Certificate & Network Boundaries       |
+---------------------------------------------------------------------------------------------------+
                                  │ Attaches Listeners
                                  ▼
+---------------------------------------------------------------------------------------------------+
| Application Developer (Product Engineers)                                                         |
|   └── HTTPRoute / GRPCRoute / TCPRoute: Deklarasi aturan routing URL, canary split, filters       |
|   └── ReferenceGrant: Izin eksplisit binding lintas-namespace secara Zero-Trust                   |
+---------------------------------------------------------------------------------------------------+

=======================================================================================================
                  CILIUM L7 GATEWAY API DATAPATH INTERNALS
=======================================================================================================
 External Client Traffic (TCP SYN to Gateway VIP)
          │
          ▼
 [ Physical Interface / Host VETH ]
          │
          ├── [ eBPF XDP Program ] (Fast L4 Drop, Anti-DDoS, Maglev Consistent Hashing)
          │
          ▼
 [ eBPF TC (Traffic Control) Ingress Filter ]
          │
          ├── Is packet destined to Gateway API IP:Port?
          │       │
          │       ▼
          │   [ Socket Redirection (sockops / sockmap) ] ──► Bypasses Kernel TCP/IP Stack!
          │       │
          │       ▼
          │   [ In-Pod Embedded Envoy Proxy Instance ] (Cilium Managed)
          │       │
          │       ├── TLS Termination (BoringSSL with Hardware Acceleration)
          │       ├── L7 Header Filtering / Rewrite / JWT Validation
          │       ├── Canary Weight Evaluation (e.g. 90% v1, 10% v2)
          │       │
          │       ▼
          │   [ Upstream Destination Selection (Backend K8s Pod) ]
          │
          ▼
 [ eBPF TC Egress Filter ] ──► Direct Memory DMA to Target Backend Pod Network Namespace

=======================================================================================================
               CGROUPS V2 MEMORY HIERARCHY CONTROLS FOR KUBERNETES PODS
=======================================================================================================
 Physical Host RAM
  └── /sys/fs/cgroup/kubepods.slice
       └── kubepods-burstable.slice
            └── kubepods-burstable-pod<UID>.slice
                 └── memory.min       : Absolute hard memory protection (never reclaimed/OOM-killed)
                 └── memory.low       : Soft protection threshold (best-effort memory reservation)
                 └── memory.high      : Throttle / Page Reclaim Boundary (K8s Request / Soft Limit)
                 └── memory.max       : Hard Out-Of-Memory (OOM) Limit -> Triggers Linux OOM Killer
                 └── memory.current   : Live memory consumption (RSS + Page Cache)

=======================================================================================================
                         KARPENTER JUST-IN-TIME PROVISIONING PIPELINE
=======================================================================================================
 [ Developer / Deployment ] ──► Pods Scaled Out (Pending Status: Unschedulable)
                                         │
                                         ▼
 ┌────────────────────────── Karpenter Controller (Node Lifecycle) ──────────────────────────────────┐
 │                                                                                                   │
 │  1. Watcher: Mendeteksi Pod dengan status 'Pending' dan Scheduler 'failedScheduling' events        │
 │  2. Topology Evaluator: Menganalisa NodeSelector, NodeAffinity, Taints, Tolerations, PVC, Topology│
 │  3. Bin-Packing Algorithm:                                                                        │
 │     - Menghitung total kebutuhan vCPU, Memory, GPU, dan Network Interface                          │
 │     - Memilih ukuran EC2 Instance paling murah & optimal (e.g. c6i.xlarge vs m6i.large Spot)       │
 │  4. Direct Cloud API Call:                                                                        │
 │     - Melakukan pemanggilan batch `ec2:RunInstances` (Fleet request) langsung ke AWS                │
 │  5. Fast Bootstrap:                                                                               │
 │     - Node booting menggunakan Bottlerocket / Ubuntu minimal image dengan pre-cached pause container│
 │  6. Kubelet Registration: Node langsung bergabung ke cluster dan Pod di-bind dalam < 45 detik      │
 │                                                                                                   │
 └───────────────────────────────────────────────────────────────────────────────────────────────────┘

# /etc/etcd/etcd.conf.yml
# Node: etcd-control-01 (IP: 198.51.100.11)
name: 'etcd-control-01'
data-dir: '/var/lib/etcd/data'
wal-dir: '/var/lib/etcd/wal' # Pisahkan WAL ke mount point NVMe terpisah jika tersedia
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 8589934592 # 8 GiB Hard Limit Storage
auto-compaction-retention: '1h'
auto-compaction-mode: 'periodic'
max-request-bytes: 1572864 # 1.5 MiB
grpc-keepalive-min-time: 5s
grpc-keepalive-interval: 2h
grpc-keepalive-timeout: 20s

# Cluster Configuration
initial-advertise-peer-urls: 'https://198.51.100.11:2380'
listen-peer-urls: 'https://198.51.100.11:2380'
listen-client-urls: 'https://198.51.100.11:2379,https://127.0.0.1:2379'
advertise-client-urls: 'https://198.51.100.11:2379'
initial-cluster: 'etcd-control-01=https://198.51.100.11:2380,etcd-control-02=https://198.51.100.12:2380,etcd-control-03=https://198.51.100.13:2380'
initial-cluster-token: 'etcd-k8s-prod-token'
initial-cluster-state: 'new'

# mTLS Peer Communication
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/peer.crt'
  key-file: '/etc/kubernetes/pki/etcd/peer.key'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.crt'

# mTLS Client-to-Server Communication
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/server.crt'
  key-file: '/etc/kubernetes/pki/etcd/server.key'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.crt'

# Logging & Monitoring
log-level: 'warn'
logger: 'zap'
log-outputs: ['stderr']
listen-metrics-urls: 'http://198.51.100.11:2381'

[Unit]
Description=etcd high-availability distributed key-value store
Documentation=https://github.com/etcd-io/etcd
After=network.target network-online.target
Wants=network-online.target

[Service]
Type=notify
User=etcd
Group=etcd
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.conf.yml
Restart=always
RestartSec=5s
LimitNOFILE=65536
LimitNPROC=65536

# Prioritas I/O Realtime untuk IOPS WAL Fsync
CPUSchedulingPolicy=rr
CPUSchedulingPriority=50
IOSchedulingClass=realtime
IOSchedulingPriority=0

# Security Hardening
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
ProtectKernelModules=true
ProtectControlGroups=true
MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: workload-critical-services
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 50
    lendablePercent: 20
    limitResponse:
      type: Queue
      queuing:
        queues: 64
        handSize: 6
        queueLengthLimit: 50
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: batch-analytics-throttled
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 5
    lendablePercent: 0
    limitResponse:
      type: Queue
      queuing:
        queues: 16
        handSize: 4
        queueLengthLimit: 10
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: protect-admission-webhooks
spec:
  priorityLevelConfiguration:
    name: workload-critical-services
  matchingPrecedence: 200
  distinguisherMethod:
    type: ByUser
  rules:
  - subjects:
    - kind: User
      user:
        name: "system:serviceaccount:kyverno:kyverno-admission-controller"
    - kind: User
      user:
        name: "system:serviceaccount:gatekeeper-system:gatekeeper-admin"
    resourceRules:
    - verbs: ["*"]
      apiGroups: ["*"]
      resources: ["*"]
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: throttle-batch-scrapers
spec:
  priorityLevelConfiguration:
    name: batch-analytics-throttled
  matchingPrecedence: 8000
  distinguisherMethod:
    type: ByUser
  rules:
  - subjects:
    - kind: Group
      group:
        name: "system:serviceaccounts:analytics"
    - kind: Group
      group:
        name: "system:serviceaccounts:reporting"
    resourceRules:
    - verbs: ["list", "get"]
      apiGroups: [""]
      resources: ["pods", "configmaps", "events"]

# GatewayClass disediakan oleh Cilium
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: cilium-gateway-class
spec:
  controllerName: io.cilium/gateway-controller
  description: "Production Cilium eBPF L7 Gateway Controller"
---
# Gateway Ingress Definition
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-external-gateway
  namespace: networking-system
spec:
  gatewayClassName: cilium-gateway-class
  listeners:
  - name: https-wildcard
    protocol: HTTPS
    port: 443
    hostname: "*.alinlab.id"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        group: ""
        name: prod-wildcard-tls-cert
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            gateway-exposure: "enabled"
---
# ReferenceGrant untuk mengizinkan Gateway di networking-system membaca Secret di security-system
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-gateway-read-tls
  namespace: security-system
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: networking-system
  to:
  - group: ""
    kind: Secret
---
# HTTPRoute: Canary Traffic Splitting & Header Manipulation
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: core-api-route
  namespace: backend-apps
spec:
  parentRefs:
  - name: prod-external-gateway
    namespace: networking-system
  hostnames:
  - "api.alinlab.id"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v2/checkout
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: X-Gateway-Routed-By
          value: "Cilium-L7-eBPF"
    backendRefs:
    - name: checkout-service-stable
      port: 8080
      weight: 90
    - name: checkout-service-canary
      port: 8080
      weight: 10

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
cgroupsPerQOS: true
cgroupRoot: /
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"
topologyManagerPolicy: single-numa-node

# Cgroups v2 Memory QoS Tuning
memoryThrottlingFactor: 0.8 # Menyetel memory.high pada 80% dari Memory Limit
failSwapOn: true
serializeImagePulls: false
maxPods: 110

# Eviction Hard Thresholds
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"

# Eviction Soft Thresholds & Grace Periods
evictionSoft:
  memory.available: "1Gi"
  nodefs.available: "15%"
evictionSoftGracePeriod:
  memory.available: "1m30s"
  nodefs.available: "2m"
evictionMaxPodGracePeriod: 60

# Node System & Kube Reserved Capacity
systemReservedCgroup: /system.slice
kubeReservedCgroup: /podruntime.slice
kubeReserved:
  cpu: "1000m"
  memory: "2Gi"
  ephemeral-storage: "3Gi"
systemReserved:
  cpu: "500m"
  memory: "1Gi"
  ephemeral-storage: "2Gi"

# PLEG and Health Parameters
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 5m
syncFrequency: 1m
fileCheckFrequency: 20s

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: compute-spot-workload
spec:
  template:
    metadata:
      labels:
        workload-tier: "stateless-api"
    spec:
      nodeClassRef:
        name: enterprise-nodeclass
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64"]
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values: ["c6i", "c7i", "m6i", "m7i"]
      - key: karpenter.k8s.aws/instance-size
        operator: In
        values: ["xlarge", "2xlarge", "4xlarge"]
      - key: topology.kubernetes.io/zone
        operator: In
        values: ["ap-southeast-1a", "ap-southeast-1b", "ap-southeast-1c"]
      disruption:
        consolidationPolicy: WhenUnderutilized
        consolidateAfter: 30s
        expireAfter: 720h # 30 Hari rolling restart
  limits:
    cpu: "1000"
    memory: "4000Gi"
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: enterprise-nodeclass
spec:
  amiFamily: Bottlerocket
  role: KarpenterNodeInstanceRole
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: "prod-eks-cluster"
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: "prod-eks-cluster"
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      volumeSize: 20Gi
      volumeType: gp3
      encrypted: true
  - deviceName: /dev/xvdb # Container runtime storage
    ebs:
      volumeSize: 100Gi
      volumeType: gp3
      iops: 4000
      throughput: 250
      encrypted: true

#!/usr/bin/env bash
# ==============================================================================
# Script: benchmark-etcd-perf.sh
# Deskripsi: Validasi & Benchmarking Performa etcd v3 Fsync Latency & Throughput
# Author: La Nina 🌊
# Status: [Riset Mandiri 🤖]
# Target: Production Kubernetes etcd Cluster (WAL fsync < 10ms threshold)
# ==============================================================================

set -euo pipefail

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
PURPLE='\033[0;35m'
CYAN='\033[0;36m'
NC='\033[0m'

echo -e "${CYAN}========================================================================${NC}"
echo -e "${PURPLE}  ⚡ ENTERPRISE ETCD STORAGE ENGINE & WAL FSYNC BENCHMARK SUITE          ${NC}"
echo -e "${PURPLE}  Author: La Nina 🌊 | Target: Tier-1 Kubernetes Control Plane          ${NC}"
echo -e "${CYAN}========================================================================${NC}"

ETCD_ENDPOINT=${ETCD_ENDPOINT:-"https://127.0.0.1:2379"}
ETCDCTL_CACERT=${ETCDCTL_CACERT:-"/etc/kubernetes/pki/etcd/ca.crt"}
ETCDCTL_CERT=${ETCDCTL_CERT:-"/etc/kubernetes/pki/etcd/server.crt"}
ETCDCTL_KEY=${ETCDCTL_KEY:-"/etc/kubernetes/pki/etcd/server.key"}

# 1. Cek etcdctl availability
if ! command -v etcdctl &>/dev/null; then
    echo -e "${YELLOW}[!] etcdctl tidak ditemukan di sistem lokal.${NC}"
    echo -e "    Mengecek apakah berjalan di dalam Pod etcd k8s..."
    if kubectl get pods -n kube-system -l component=etcd &>/dev/null; then
        ETCD_POD=$(kubectl get pods -n kube-system -l component=etcd -o jsonpath='{.items[0].metadata.name}')
        echo -e "    ${GREEN}✓ Ditemukan etcd pod: $ETCD_POD${NC}"
        RUN_IN_POD=true
    else
        echo -e "${RED}[ERROR] Tidak dapat menemukan etcdctl ataupun Kubernetes etcd pod!${NC}"
        exit 1
    fi
else
    RUN_IN_POD=false
fi

echo -e "\n${BLUE}[1/4] Memeriksa Status Kesehatan Cluster & Raft Leadership...${NC}"
if [ "$RUN_IN_POD" = true ]; then
    kubectl exec -n kube-system "$ETCD_POD" -- sh -c "ETCDCTL_API=3 etcdctl --cacert=$ETCDCTL_CACERT --cert=$ETCDCTL_CERT --key=$ETCDCTL_KEY --endpoints=$ETCD_ENDPOINT endpoint health"
    echo -e "\n${BLUE}Member Status & Database Size:${NC}"
    kubectl exec -n kube-system "$ETCD_POD" -- sh -c "ETCDCTL_API=3 etcdctl --cacert=$ETCDCTL_CACERT --cert=$ETCDCTL_CERT --key=$ETCDCTL_KEY --endpoints=$ETCD_ENDPOINT endpoint status -w table"
else
    ETCDCTL_API=3 etcdctl --cacert="$ETCDCTL_CACERT" --cert="$ETCDCTL_CERT" --key="$ETCDCTL_KEY" --endpoints="$ETCD_ENDPOINT" endpoint health
    echo -e "\n${BLUE}Member Status & Database Size:${NC}"
    ETCDCTL_API=3 etcdctl --cacert="$ETCDCTL_CACERT" --cert="$ETCDCTL_CERT" --key="$ETCDCTL_KEY" --endpoints="$ETCD_ENDPOINT" endpoint status -w table
fi

echo -e "\n${BLUE}[2/4] Menguji Disk Fsync Latency menggunakan fio (Simulasi etcd WAL sequential 8K writes)...${NC}"
if command -v fio &>/dev/null; then
    FIO_DIR="/var/lib/etcd-test-wal"
    mkdir -p "$FIO_DIR"
    
    echo -e "  - Menjalankan I/O write stress test dengan O_SYNC..."
    fio --name=etcd-wal-bench --filename="$FIO_DIR/wal.bin" --size=100M --bs=8k \
        --ioengine=sync --rw=write --fsync=1 --direct=1 --numjobs=1 --runtime=15 --time_based \
        --output="/tmp/fio-etcd.log"
    
    # Ekstraksi f_sync 99th percentile
    FSYNC_P99=$(grep -A 5 "fsync" /tmp/fio-etcd.log | grep "99.00th" | awk -F'=' '{print $2}' | tr -d ' ' || echo "N/A")
    echo -e "  - Hasil Fsync p99: ${YELLOW}${FSYNC_P99}${NC}"
    rm -rf "$FIO_DIR" /tmp/fio-etcd.log
else
    echo -e "  ${YELLOW}[!] Tools 'fio' tidak ditemukan. Lewati tes benchmark disk mentah.${NC}"
fi

echo -e "\n${BLUE}[3/4] Melakukan Benchmark Write Throughput etcd (etcdctl check perf)...${NC}"
if [ "$RUN_IN_POD" = true ]; then
    kubectl exec -n kube-system "$ETCD_POD" -- sh -c "ETCDCTL_API=3 etcdctl --cacert=$ETCDCTL_CACERT --cert=$ETCDCTL_CERT --key=$ETCDCTL_KEY --endpoints=$ETCD_ENDPOINT check perf" || true
else
    ETCDCTL_API=3 etcdctl --cacert="$ETCDCTL_CACERT" --cert="$ETCDCTL_CERT" --key="$ETCDCTL_KEY" --endpoints="$ETCD_ENDPOINT" check perf || true
fi

echo -e "\n${BLUE}[4/4] Analisis Ambang Batas Toleransi Produksi:${NC}"
echo -e "  - Ambang batas WAL fsync delay: ${GREEN}< 10 ms${NC} (Hard limit: > 50 ms memicu election timeout / leader loss)"
echo -e "  - Ambang batas backend commit delay: ${GREEN}< 25 ms${NC}"
echo -e "  - Rekomendasi Storage: NVMe Enterprise (DWPD >= 3), dedicated disk terpisah dari root OS."

echo -e "\n${GREEN}[✓] Benchmark & Validasi etcd selesai! 🫡🌊${NC}"

#!/usr/bin/env bash
# ==============================================================================
# Script: benchmark-k8s-apiserver-apf.sh
# Deskripsi: Validasi & Benchmarking Kube-Apiserver API Priority and Fairness (APF)
# Author: La Nina 🌊
# Status: [Riset Mandiri 🤖]
# Target: Kubernetes v1.30+ Enterprise Control Plane
# ==============================================================================

set -euo pipefail

# Warna Output Terminal
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
PURPLE='\033[0;35m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color

echo -e "${CYAN}========================================================================${NC}"
echo -e "${PURPLE}  🚀 KUBE-APISERVER API PRIORITY & FAIRNESS (APF) BENCHMARK SUITE       ${NC}"
echo -e "${PURPLE}  Author: La Nina 🌊 | Target: Tier-1 Kubernetes Production             ${NC}"
echo -e "${CYAN}========================================================================${NC}"

# Validasi Dependensi
for cmd in kubectl curl jq bc parallel; do
    if ! command -v "$cmd" &> /dev/null; then
        echo -e "${YELLOW}[!] Command '$cmd' tidak ditemukan. Menggunakan fallback standar jika memungkinkan...${NC}"
    fi
done

# Parameter Konfigurasi Benchmark
CONCURRENCY=${CONCURRENCY:-50}
TOTAL_REQUESTS=${TOTAL_REQUESTS:-500}
TARGET_NAMESPACE=${TARGET_NAMESPACE:-"default"}
OUTPUT_DIR="/tmp/apf-benchmark-$(date +%s)"
mkdir -p "$OUTPUT_DIR"

echo -e "\n${BLUE}[1/4] Mengumpulkan Informasi Kube-Apiserver & Konfigurasi APF...${NC}"
SERVER_VERSION=$(kubectl version --client=false -o json | jq -r '.serverVersion.gitVersion')
echo -e "  - K8s Server Version: ${GREEN}${SERVER_VERSION}${NC}"

echo -e "\n${BLUE}[2/4] Memeriksa PriorityLevelConfigurations & FlowSchemas Aktif:${NC}"
printf "%-30s %-15s %-15s %-15s\n" "FLOW SCHEMA" "PRIORITY LEVEL" "MATCHING RULES" "STATUS"
echo "----------------------------------------------------------------------------"
kubectl get flowschemas.flowcontrol.apiserver.k8s.io -o json | jq -r '.items[] | [.metadata.name, .spec.priorityLevelConfiguration.name, (.spec.rules | length | tostring)] | @tsv' | while IFS=$'\t' read -r fs pl rules; do
    printf "%-30s %-15s %-15s %-15s\n" "$fs" "$pl" "$rules" "${GREEN}Active${NC}"
done

echo -e "\n${BLUE}[3/4] Mengambil Baseline Metrik APF dari Prometheus Endpoint Kube-Apiserver...${NC}"
METRICS_TMP="$OUTPUT_DIR/apf_metrics_initial.txt"
if kubectl get --raw /metrics > "$METRICS_TMP" 2>/dev/null; then
    echo -e "  ${GREEN}✓ Berhasil mengikis /metrics dari kube-apiserver.${NC}"
    REJECTED_BEFORE=$(grep -E 'apiserver_flowcontrol_rejected_requests_total' "$METRICS_TMP" | grep -v '#' | awk '{sum+=$NF} END {print sum+0}')
    QUEUED_BEFORE=$(grep -E 'apiserver_flowcontrol_current_inqueue_requests' "$METRICS_TMP" | grep -v '#' | awk '{sum+=$NF} END {print sum+0}')
    EXECUTING_BEFORE=$(grep -E 'apiserver_flowcontrol_current_executing_requests' "$METRICS_TMP" | grep -v '#' | awk '{sum+=$NF} END {print sum+0}')
    echo -e "  - Initial Rejected Requests : ${YELLOW}${REJECTED_BEFORE}${NC}"
    echo -e "  - Initial In-Queue Requests : ${YELLOW}${QUEUED_BEFORE}${NC}"
    echo -e "  - Initial Executing Seats   : ${YELLOW}${EXECUTING_BEFORE}${NC}"
else
    echo -e "  ${YELLOW}[!] Endpoint /metrics memerlukan otentikasi cluster-admin direct. Melanjutkan uji beban synthetic...${NC}"
fi

echo -e "\n${BLUE}[4/4] Menjalankan Uji Beban Konkurensi Tinggi (Simulasi APF Throttle)...${NC}"
echo -e "  - Target Request : Pod LIST/GET operations via API"
echo -e "  - Concurrency    : ${CONCURRENCY} worker threads"
echo -e "  - Total Requests : ${TOTAL_REQUESTS} requests"

START_TIME=$(date +%s%N)

run_worker_batch() {
    local worker_id=$1
    local req_per_worker=$2
    local out_log="$OUTPUT_DIR/worker_${worker_id}.log"
    
    for ((i=1; i<=req_per_worker; i++)); do
        local start_req
        local end_req
        local duration
        local http_code
        
        start_req=$(date +%s%N)
        # Eksekusi raw kubectl get pod dengan output minimal
        http_code=$(kubectl get pods -n "$TARGET_NAMESPACE" --limit=1 -o name >/dev/null 2>&1 && echo "200" || echo "429_or_ERR")
        end_req=$(date +%s%N)
        
        duration=$(( (end_req - start_req) / 1000000 )) # milliseconds
        echo "$http_code $duration" >> "$out_log"
    done
}

export -f run_worker_batch
export TARGET_NAMESPACE
export OUTPUT_DIR

REQS_PER_WORKER=$(( TOTAL_REQUESTS / CONCURRENCY ))
pids=()

for ((w=1; w<=CONCURRENCY; w++)); do
    run_worker_batch "$w" "$REQS_PER_WORKER" &
    pids+=($!)
done

# Tunggu seluruh worker selesai
for pid in "${pids[@]}"; do
    wait "$pid"
done

END_TIME=$(date +%s%N)
TOTAL_DURATION_SEC=$(echo "scale=3; ($END_TIME - $START_TIME) / 1000000000" | bc -l)

echo -e "\n${CYAN}========================================================================${NC}"
echo -e "${GREEN}                 HASIL BENCHMARK & EVALUASI APF                         ${NC}"
echo -e "${CYAN}========================================================================${NC}"

# Agregasi data hasil
COMBINED_LOG="$OUTPUT_DIR/all_requests.log"
cat "$OUTPUT_DIR"/worker_*.log > "$COMBINED_LOG"

TOTAL_SUCCESS=$(grep -c "^200" "$COMBINED_LOG" || true)
TOTAL_THROTTLED=$(grep -vc "^200" "$COMBINED_LOG" || true)
RPS=$(echo "scale=2; $TOTAL_REQUESTS / $TOTAL_DURATION_SEC" | bc -l)

# Ekstraksi Latency Percentiles (ms)
LATENCIES=($(awk '{print $2}' "$COMBINED_LOG" | sort -n))
COUNT=${#LATENCIES[@]}

P50_INDEX=$(( COUNT * 50 / 100 ))
P90_INDEX=$(( COUNT * 90 / 100 ))
P99_INDEX=$(( COUNT * 99 / 100 ))

P50=${LATENCIES[$P50_INDEX]:-0}
P90=${LATENCIES[$P90_INDEX]:-0}
P99=${LATENCIES[$P99_INDEX]:-0}

echo -e "Durasi Eksekusi Total : ${YELLOW}${TOTAL_DURATION_SEC} detik${NC}"
echo -e "Throughput Aktual     : ${GREEN}${RPS} req/sec${NC}"
echo -e "Total Requests        : ${TOTAL_REQUESTS}"
echo -e "Request Sukses (200)  : ${GREEN}${TOTAL_SUCCESS}${NC}"
echo -e "Throttled/Drop (429)  : ${RED}${TOTAL_THROTTLED}${NC}"
echo -e "Latency Distribution  :"
echo -e "  - p50 (Median)      : ${CYAN}${P50} ms${NC}"
echo -e "  - p90               : ${CYAN}${P90} ms${NC}"
echo -e "  - p99 (Tail Latency): ${RED}${P99} ms${NC}"

echo -e "\n${BLUE}[*] Rekomendasi Tuning APF:${NC}"
if [ "$TOTAL_THROTTLED" -gt 0 ]; then
    echo -e "  ${YELLOW}[!] Terdeteksi request rejection/throttling ($TOTAL_THROTTLED). Tingkatkan 'assuredConcurrencyShares' atau 'queuing.queueLengthLimit' pada PriorityLevelConfiguration sasaran.${NC}"
else
    echo -e "  ${GREEN}✓ Tidak ada request dropped. Fair queuing dan concurrency seats APF berjalan optimal dalam ambang batas konkurensi $CONCURRENCY.${NC}"
fi

# Cleanup
rm -rf "$OUTPUT_DIR"
echo -e "\n${GREEN}[✓] Benchmark APF selesai sempurna! 🫡🌊${NC}"

#!/usr/bin/env bash
# ==============================================================================
# Script: audit-gateway-api-cilium.sh
# Deskripsi: Diagnostic & Compliance Auditor Kubernetes Gateway API + Cilium L7
# Author: La Nina 🌊
# Status: [Riset Mandiri 🤖]
# Target: Kubernetes v1.30+ Enterprise Gateway API
# ==============================================================================

set -euo pipefail

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
PURPLE='\033[0;35m'
CYAN='\033[0;36m'
NC='\033[0m'

echo -e "${CYAN}========================================================================${NC}"
echo -e "${PURPLE}  🌐 KUBERNETES GATEWAY API & CILIUM L7 AUDITOR & DIAGNOSTIC           ${NC}"
echo -e "${PURPLE}  Author: La Nina 🌊 | Target: Tier-1 Production Cloud-Native Mesh       ${NC}"
echo -e "${CYAN}========================================================================${NC}"

# 1. Cek CRD Gateway API
echo -e "\n${BLUE}[1/5] Verifikasi Instalasi Gateway API Custom Resource Definitions (CRDs)...${NC}"
CRDS=(
    "gatewayclasses.gateway.networking.k8s.io"
    "gateways.gateway.networking.k8s.io"
    "httproutes.gateway.networking.k8s.io"
    "grpcroutes.gateway.networking.k8s.io"
    "referencegrants.gateway.networking.k8s.io"
)

CRD_MISSING=0
for crd in "${CRDS[@]}"; do
    if kubectl get crd "$crd" &>/dev/null; then
        echo -e "  - CRD ${GREEN}$crd${NC} terinstal."
    else
        echo -e "  - CRD ${RED}$crd${NC} [TIDAK DITEMUKAN]!"
        CRD_MISSING=$((CRD_MISSING + 1))
    fi
done

if [ "$CRD_MISSING" -gt 0 ]; then
    echo -e "  ${YELLOW}[!] Gateway API CRD belum lengkap. Pasang via standard channel:${NC}"
    echo -e "      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml"
fi

# 2. Status GatewayClass
echo -e "\n${BLUE}[2/5] Memeriksa Status GatewayClass...${NC}"
if kubectl get gatewayclasses &>/dev/null; then
    kubectl get gatewayclasses -o custom-columns="NAME:.metadata.name,CONTROLLER:.spec.controllerName,ACCEPTED:.status.conditions[?(@.type=='Accepted')].status,DESCRIPTION:.spec.description"
else
    echo -e "  ${YELLOW}[!] Tidak ada GatewayClass yang didefinisikan.${NC}"
fi

# 3. Status Gateway Instances & Listener Health
echo -e "\n${BLUE}[3/5] Memeriksa Gateway Instance & Listeners...${NC}"
if kubectl get gateways -A &>/dev/null; then
    printf "%-15s %-25s %-12s %-12s %-20s\n" "NAMESPACE" "GATEWAY" "PROGRAMMED" "LISTENERS" "ADDRESSES"
    echo "---------------------------------------------------------------------------------------"
    kubectl get gateways -A -o json | jq -r '.items[] | [.metadata.namespace, .metadata.name, (.status.conditions[]? | select(.type=="Programmed").status // "False"), (.spec.listeners | length | tostring), (([.status.addresses[]?.value] | join(",")) // "Pending")] | @tsv' | while IFS=$'\t' read -r ns gw prog list_count addrs; do
        if [ "$prog" == "True" ]; then
            P_COLOR="${GREEN}True${NC}"
        else
            P_COLOR="${RED}${prog}${NC}"
        fi
        printf "%-15s %-25s %-21b %-12s %-20s\n" "$ns" "$gw" "$P_COLOR" "$list_count" "$addrs"
    done
else
    echo -e "  ${YELLOW}[!] Tidak ada Gateway terpasang.${NC}"
fi

# 4. Status HTTPRoutes & ParentRefs Attachment
echo -e "\n${BLUE}[4/5] Memeriksa HTTPRoutes & Backend Service Attachment...${NC}"
if kubectl get httproutes -A &>/dev/null; then
    printf "%-15s %-25s %-25s %-15s\n" "NAMESPACE" "HTTPROUTE" "PARENT GATEWAY" "ACCEPTED STATUS"
    echo "---------------------------------------------------------------------------------------"
    kubectl get httproutes -A -o json | jq -r '.items[] | [.metadata.namespace, .metadata.name, (([.spec.parentRefs[]?.name] | join(",")) // "None"), (.status.parents[0]?.conditions[]? | select(.type=="Accepted").status // "False")] | @tsv' | while IFS=$'\t' read -r ns rt parent acc; do
        if [ "$acc" == "True" ]; then
            A_COLOR="${GREEN}True${NC}"
        else
            A_COLOR="${RED}${acc}${NC}"
        fi
        printf "%-15s %-25s %-25s %-24b\n" "$ns" "$rt" "$parent" "$A_COLOR"
    done
else
    echo -e "  ${YELLOW}[!] Tidak ada HTTPRoute terpasang.${NC}"
fi

# 5. ReferenceGrant Security Audit (Cross-Namespace Security)
echo -e "\n${BLUE}[5/5] Memeriksa Kebijakan Zero-Trust Cross-Namespace (ReferenceGrant)...${NC}"
if kubectl get referencegrants -A &>/dev/null; then
    RG_COUNT=$(kubectl get referencegrants -A --no-headers | wc -l)
    echo -e "  - Ditemukan ${GREEN}$RG_COUNT${NC} ReferenceGrant policies."
    kubectl get referencegrants -A -o custom-columns="NAMESPACE:.metadata.name,FROM_KIND:.spec.from[0].kind,FROM_NS:.spec.from[0].namespace,TO_KIND:.spec.to[0].kind"
else
    echo -e "  - ${YELLOW}Tidak ada ReferenceGrant ditemukan. Gateway tidak diizinkan mereferensikan Secret/Backend di namespace lain secara Zero-Trust.${NC}"
fi

echo -e "\n${GREEN}[✓] Audit Gateway API & Cilium L7 selesai! 🫡🌊${NC}"

# Log dari etcd-control-02 (Follower):
2026-09-13T10:14:02.129Z WARN rafthttp: failed to find member 4a8e291b in cluster or it is dead
2026-09-13T10:14:02.341Z INFO raft: 8c72b10a [term: 42] received a MsgVote message from 8c72b10a...
2026-09-13T10:14:02.341Z INFO raft: 8c72b10a became candidate at term 43
2026-09-13T10:14:02.512Z INFO raft: 8c72b10a elected leader at term 43

# Log dari etcd-control-01 (Mantan Leader):
2026-09-13T10:14:01.980Z WARN wal: sync duration of 2.418291s, expected less than 10ms
2026-09-13T10:14:02.550Z INFO raft: 4a8e291b [term: 42] received a MsgApp message with higher term from 8c72b10a [term: 43]
2026-09-13T10:14:02.550Z INFO raft: 4a8e291b became follower at term 43

level=warning msg="Cannot forward L7 traffic: Backend service has no valid endpoints" 
controller=cilium-gateway-controller httproute=backend-apps/core-api-route 
backendRef=checkout-service-stable error="EndpointsNotFound"

# Memeriksa file cgroup pod di host:
cat /sys/fs/cgroup/kubepods.slice/.../memory.current
# Hasil: 4180000000 (~3.98 GiB)

cat /sys/fs/cgroup/kubepods.slice/.../memory.high
# Hasil: 3435973836 (~3.2 GiB)

cat /sys/fs/cgroup/kubepods.slice/.../memory.events
# Hasil:
high 482109
max 0
oom 0
oom_kill 0