Arsitektur Carrier-Grade Cloud-Native Networking Skala Enterprise: Deep-Dive DPDK PMD, FD.io VPP Vector Engine, SR-IOV VFIO, dan Cilium ClusterMesh WireGuard

+---------------------------------------------------------------------------------------------------+
| 100 GbE Wire (148.8 Mpps, 64B Frames) -> 6.72 ns Budget per Packet (~20 CPU Cycles at 3.0 GHz)   |
+---------------------------------------------------------------------------------------------------+
| Linux Kernel Standard Socket (sk_buff)  : 1,500 - 3,500 cycles/pkt -> Max 1 - 2 Mpps per core    |
| Linux eBPF / XDP (Driver / Native Mode) : 100 - 250 cycles/pkt     -> Max 15 - 30 Mpps per core   |
| DPDK / VPP Kernel-Bypass (Vector Engine): 15 - 45 cycles/pkt       -> Line-Rate 50 - 148 Mpps     |
+---------------------------------------------------------------------------------------------------+

========================================================================================
             TRADITIONAL LINUX KERNEL DATAPATH vs DPDK KERNEL-BYPASS
========================================================================================

--- [ KERNEL DATAPATH KONVENSIONAL ] ---
  1. Wire -> NIC Hardware RX FIFO Ring Buffer
  2. NIC memicu Hardware Interrupt (MSI-X) -> Suspensi eksekusi core CPU
  3. ISR (Interrupt Service Routine) acknowledge IRQ & jadwalkan NET_RX_SOFTIRQ
  4. Context Switch ke ksoftirqd / NAPI poll loop
  5. kmalloc() dinamis untuk struct sk_buff (~240 byte metadata overhead per frame)
  6. DMA transfer dari NIC RX ring buffer ke host memory kernel space
  7. Traversal rantai Netfilter/Iptables/Conntrack (pencarian linier bertingkat)
  8. Routing table lookup (L3 FIB trie lookup) & alokasi socket receive queue
  9. Context Switch: proses userspace terbangun via epoll_wait() (Ring 0 -> Ring 3)
 10. copy_to_user(): CPU menyalin memori dari kernel sk_buff ke userspace memory
 11. kfree_skb(): Kernel dealokasi struktur memori, memicu fragmentasi slab allocator

--- [ DPDK KERNEL-BYPASS DATAPATH ] ---
  1. Wire -> NIC Hardware RX Descriptor Ring (Direct Memory Access via IOMMU)
  2. Poll Mode Driver (PMD) membaca descriptor langsung dari userspace ring tanpa IRQ
  3. Pre-allocated rte_mempool: Memory buffer sudah dipersiapkan di 1GB HugePages
  4. Zero-Copy Pointer Dereference: Tidak ada duplikasi memori antar privilege ring
  5. Dedicated Isolated Core: Worker thread terkunci permanen di Ring 3 (No Context Switch)

+---------------------------------------------------------------------------------------------------+
|                                FD.io VPP DIRECTED GRAPH ARCHITECTURE                              |
+---------------------------------------------------------------------------------------------------+
|  Vektor Paket (Array hingga 256 paket per batch)                                                 |
|                                                                                                   |
|    +-------------------+       +-------------------+       +-------------------+                  |
|    |    dpdk-input     | ----> |  ethernet-input   | ----> |     ip4-input     |                  |
|    +-------------------+       +-------------------+       +-------------------+                  |
|     (Batch DMA Poll)            (L2 Header Parse)           (IPv4 Header Verify)                  |
|                                                                      |                            |
|                                                                      v                            |
|    +-------------------+       +-------------------+       +-------------------+                  |
|    | interface-output  | <---- |    ip4-rewrite    | <---- |    ip4-lookup     |                  |
|    +-------------------+       +-------------------+       +-------------------+                  |
|     (Hardware TX DMA)           (MAC Rewriting)             (FIB Trie Lookup)                     |
+---------------------------------------------------------------------------------------------------+

+---------------------------------------------------------------------------------------------------+
|                                 SR-IOV HARDWARE VIRTUALIZATION ARCHITECTURE                       |
+---------------------------------------------------------------------------------------------------+
|                                PHYSICAL PCIe NIC (e.g. Intel E810 100GbE)                         |
|  +---------------------------------------------------------------------------------------------+  |
|  | Physical Function (PF): PCI Management, Firmware, Global Link Configuration (ens1f0)        |  |
|  +---------------------------------------------------------------------------------------------+  |
|        |                  |                  |                  |                  |              |
|        +--------+---------+--------+---------+--------+---------+--------+---------+              |
|                 |                  |                  |                  |                        |
|                 v                  v                  v                  v                        |
|             [ VF 0 ]           [ VF 1 ]           [ VF 2 ]           [ VF 3 ]                     |
|                 |                  |                  |                  |                        |
|            (vfio-pci)         (vfio-pci)         (iavf driver)      (iavf driver)                 |
|                 v                  v                  v                  v                        |
|          IOMMU Group 42     IOMMU Group 43      Linux Kernel       Linux Kernel                   |
|                 |                  |             (Host eth1)       (K8s Default)                  |
|                 v                  v                                                              |
|          K8s Telco Pod 1    K8s Telco Pod 2                                                       |
|          (VPP / DPDK UPF)   (BGP Edge Router)                                                     |
|          Zero-Copy DMA      Zero-Copy DMA                                                         |
+---------------------------------------------------------------------------------------------------+

# Production GRUB Kernel Parameters for DPDK & High-Throughput Networking
# Hardware: Dual-Socket Intel Xeon Scalable (32 Cores per Socket, 64 Cores Total)
# Cores 0-3, 32-35: Dialokasikan untuk Host OS, Kubelet, dan Systemd Daemons
# Cores 4-31, 36-63: Terisolasi khusus untuk DPDK / VPP Worker Threads
# Hugepages: 64 Hugepages berukuran 1GB (Total 64GB Alokasi Memori Hugepage)

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT \
intel_iommu=on \
iommu=pt \
default_hugepagesz=1G \
hugepagesz=1G \
hugepages=64 \
isolcpus=managed_irq,domain,4-31,36-63 \
nohz_full=4-31,36-63 \
rcu_nocbs=4-31,36-63 \
irqaffinity=0-3,32-35 \
tsc=reliable \
clocksource=tsc \
processor.max_cstate=1 \
intel_idle.max_cstate=0 \
intel_pstate=performance \
transparent_hugepage=never \
nosoftlockup"

# ==============================================================================
# Linux Carrier-Grade High-Throughput Network Engine Tuning
# Production parameters for 10GbE / 40GbE / 100GbE Infrastructure
# ==============================================================================

# Alokasi buffer socket maksimum (64MB read/write buffers)
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.core.rmem_default = 33554432
net.core.wmem_default = 33554432
net.core.optmem_max = 2048576

# TCP Buffer Autotuning: min, default, max
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Batas antrian device backlog untuk mencegah drop paket saat burst traffic
net.core.netdev_max_backlog = 250000
net.core.netdev_budget = 600
net.core.netdev_budget_usecs = 4000

# Somaxconn connection backlog queue
net.core.somaxconn = 65535

# Algoritma Congestion Control: BBRv3 dengan Fair Queueing (FQ)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Efisiensi koneksi TCP
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5

# Forwarding & Anti-Spoofing
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0

# Limit file & memory mapping
fs.file-max = 20971520
fs.nr_open = 20971520
vm.max_map_count = 1048576
vm.swappiness = 0

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: intel-e810-dpdk-policy
  namespace: sriov-network-operator
spec:
  resourceName: intel_e810_vfio
  nodeSelector:
    node-role.kubernetes.io/worker-telco: ""
  priority: 99
  numVfs: 8
  nicSelector:
    vendor: "8086"
    deviceID: "1592" # Intel Columbiaville E810-C 100GbE
    pfNames: ["ens1f0"]
  deviceType: vfio-pci
  isRdma: false

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: sriov-telco-data-plane
  namespace: sriov-network-operator
spec:
  resourceName: intel_e810_vfio
  networkNamespace: telco-production
  ipam: |
    {
      "type": "static",
      "addresses": [
        {
          "address": "198.51.100.10/24",
          "gateway": "198.51.100.1"
        }
      ]
    }
  vlan: 100
  spoofChk: "off"
  trust: "on"

apiVersion: v1
kind: Pod
metadata:
  name: vpp-carrier-upf-01
  namespace: telco-production
  annotations:
    k8s.v1.cni.cncf.io/networks: |
      [
        {
          "name": "sriov-telco-data-plane",
          "interface": "dpdk0"
        }
      ]
spec:
  nodeSelector:
    node-role.kubernetes.io/worker-telco: ""
  containers:
  - name: upf-engine
    image: registry.enterprise.internal/telco/vpp-upf:25.02-dpdk
    imagePullPolicy: IfNotPresent
    command: ["/usr/bin/vpp", "-c", "/etc/vpp/startup.conf"]
    securityContext:
      privileged: false
      capabilities:
        add:
          - IPC_LOCK
          - SYS_RAWIO
          - NET_ADMIN
          - SYS_ADMIN
    resources:
      requests:
        memory: "16Gi"
        cpu: "8"
        hugepages-1Gi: "16Gi"
        openshift.io/intel_e810_vfio: "1"
      limits:
        memory: "16Gi"
        cpu: "8"
        hugepages-1Gi: "16Gi"
        openshift.io/intel_e810_vfio: "1"
    volumeMounts:
    - mountPath: /dev/hugepages
      name: hugepage-1gi
    - mountPath: /etc/vpp
      name: vpp-config
  volumes:
  - name: hugepage-1gi
    emptyDir:
      medium: HugePages-1Gi
  - name: vpp-config
    configMap:
      name: vpp-startup-config

unix {
    nodaemon
    log /var/log/vpp/vpp.log
    full-coredump
    cli-listen /run/vpp/cli.sock
    gid vpp
}

memory {
    main-heap-size 4G
    main-heap-page-size 1G
}

cpu {
    main-core 4
    corelist-workers 5-8
    scheduler-policy fifo
    scheduler-priority 50
}

buffers {
    buffers-per-numa 256000
    default data-size 2048
    page-size 1G
}

dpdk {
    dev 0000:18:02.0 {
        name dpdk0
        num-rx-queues 4
        num-tx-queues 4
        num-rx-desc 2048
        num-tx-desc 2048
        tso on
        rss on
    }
    socket-mem 8192,0
    no-tx-checksum-offload
    uio-driver vfio-pci
}

plugins {
    plugin default { enable }
    plugin dpdk_plugin.so { enable }
    plugin upf_plugin.so { enable }
    plugin nat_plugin.so { disable }
}

+---------------------------------------------------------------------------------------------------+
|                                CILIUM CLUSTERMESH MULTI-DATACENTER ARCHITECTURE                    |
+---------------------------------------------------------------------------------------------------+
|  [ DATACENTER A - Jakarta (Cluster ID: 1) ]        [ DATACENTER B - Singapore (Cluster ID: 2) ]   |
|                                                                                                   |
|   +-------------------------------+                 +-------------------------------+             |
|   | Pod A (10.244.1.15)           |                 | Pod B (10.245.2.20)           |             |
|   +-------------------------------+                 +-------------------------------+             |
|         |                                                 ^                                       |
|         v                                                 |                                       |
|   Cilium eBPF Datapath                              Cilium eBPF Datapath                          |
|   (Direct Socket Delivery)                          (Direct Socket Delivery)                      |
|         |                                                 ^                                       |
|         v                                                 |                                       |
|   Node A Gateway (192.0.2.10)                       Node B Gateway (198.51.100.20)                |
|         |                                                 ^                                       |
|         +============== [ High-Throughput WireGuard Mesh ] +                                       |
|         |               Encapsulation: WireGuard ChaCha20 |                                       |
|         |               MTU Clamped to 1420 Bytes         |                                       |
|         v                                                 v                                       |
|   +-------------------------------+                 +-------------------------------+             |
|   | Top-of-Rack Switch (AS 65001) | <--- eBGP --->  | Top-of-Rack Switch (AS 65002) |             |
|   +-------------------------------+                 +-------------------------------+             |
+---------------------------------------------------------------------------------------------------+

cluster:
  name: "dc01-jkt-k8s"
  id: 1

# Kube-proxy replacement penuh berbasis eBPF
kubeProxyReplacement: "true"
k8sServiceHost: "192.0.2.100"
k8sServicePort: 6443

bpf:
  masquerade: true
  tproxy: true
  preallocateMaps: true
  mapSizeConntrack: 1048576
  mapSizeNatGlobal: 1048576
  mapSizePolicy: 131072
  sockops:
    enabled: true

# Direct Server Return (DSR) Maglev Hashing
loadBalancer:
  mode: "dsr"
  algorithm: "maglev"

# Enkripsi Transparan Cross-Cluster WireGuard
encryption:
  enabled: true
  type: wireguard
  wireguard:
    userspaceFallback: false

# Arsitektur ClusterMesh
clustermesh:
  useAPIServer: true
  apiserver:
    service:
      type: NodePort
      nodePort: 32379
    replicas: 3
    tls:
      auto:
        method: helm
  config:
    enabled: true
    clusters:
      - name: "dc02-sin-k8s"
        address: "198.51.100.10"
        port: 32379
        ips:
          - "198.51.100.11"
          - "198.51.100.12"
          - "198.51.100.13"

# BGP Control Plane Terintegrasi
bgpControlPlane:
  enabled: true

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: tor-leaf-peering
spec:
  nodeSelectors:
    matchLabels:
      rack: rack-01
  virtualRouters:
    - localASN: 65001
      exportPodCIDR: true
      neighbors:
        - peerAddress: "192.0.2.1/32"
          peerASN: 65000
          eBGPMultihopTTL: 4
          connectRetryTimeSeconds: 10
          holdTimeSeconds: 30
          keepaliveTimeSeconds: 10
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120
      serviceSelector:
        matchExpressions:
          - {key: "service.cilium.io/bgp-anycast", operator: In, values: ["true"]}

#!/usr/bin/env bash
# ==============================================================================
# Script: validate-sriov-dpdk-health.sh
# Auditor Kesiapan Host Linux untuk SR-IOV, IOMMU, HugePages & DPDK
# ==============================================================================
set -eo pipefail

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

echo -e "${BLUE}===================================================================${NC}"
echo -e "${BLUE}     SR-IOV, IOMMU & DPDK HIGH-THROUGHPUT SYSTEM AUDITOR          ${NC}"
echo -e "${BLUE}===================================================================${NC}"

ERRORS=0
WARNINGS=0

# 1. Audit Status Kernel IOMMU
echo -e "\n${YELLOW}[1/5] Memeriksa Status Kernel IOMMU...${NC}"
if [ -d "/sys/kernel/iommu_groups" ]; then
    TOTAL_GROUPS=$(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d | wc -l)
    echo -e "${GREEN}[OK] IOMMU aktif dengan $TOTAL_GROUPS IOMMU groups terisolasi.${NC}"
else
    echo -e "${RED}[ERROR] IOMMU TIDAK aktif! Aktifkan intel_iommu=on iommu=pt pada GRUB.${NC}"
    ERRORS=$((ERRORS + 1))
fi

# 2. Audit Alokasi 1GB Hugepages
echo -e "\n${YELLOW}[2/5] Memeriksa Alokasi 1GB HugePages...${NC}"
TOTAL_1G=$(grep -i HugePages_Total /proc/meminfo | awk '{print $2}' || echo 0)
FREE_1G=$(grep -i HugePages_Free /proc/meminfo | awk '{print $2}' || echo 0)
PAGE_SIZE=$(grep -i Hugepagesize /proc/meminfo | awk '{print $2, $3}')

echo -e "  - Page Size Default : ${BLUE}$PAGE_SIZE${NC}"
echo -e "  - Total HugePages   : ${BLUE}$TOTAL_1G${NC}"
echo -e "  - Free HugePages    : ${GREEN}$FREE_1G${NC}"

if [ "$TOTAL_1G" -eq 0 ]; then
    echo -e "${RED}[ERROR] Tidak ada HugePages yang dialokasikan! DPDK memerlukan HugePages.${NC}"
    ERRORS=$((ERRORS + 1))
fi

# 3. Audit Isolasi Core CPU
echo -e "\n${YELLOW}[3/5] Memeriksa Konfigurasi Isolasi Core CPU...${NC}"
CMDLINE=$(cat /proc/cmdline)
ISOLCPUS=$(echo "$CMDLINE" | grep -o 'isolcpus=[^ ]*' || echo "None")
NOHZ_FULL=$(echo "$CMDLINE" | grep -o 'nohz_full=[^ ]*' || echo "None")

echo -e "  - $ISOLCPUS"
echo -e "  - $NOHZ_FULL"

if [ "$ISOLCPUS" = "None" ]; then
    echo -e "${YELLOW}[WARN] Parameter isolcpus belum aktif. Thread PMD akan rentan context switch.${NC}"
    WARNINGS=$((WARNINGS + 1))
else
    echo -e "${GREEN}[OK] Core CPU terisolasi dari penjadwalan standar.${NC}"
fi

# 4. Audit Driver VFIO
echo -e "\n${YELLOW}[4/5] Memeriksa Modul Driver VFIO...${NC}"
if lsmod | grep -q "vfio_pci"; then
    echo -e "${GREEN}[OK] Modul vfio_pci dimuat di kernel.${NC}"
else
    echo -e "${YELLOW}[WARN] Modul vfio_pci belum dimuat. Jalankan: modprobe vfio-pci.${NC}"
    WARNINGS=$((WARNINGS + 1))
fi

# 5. Hitung Virtual Functions Aktif
echo -e "\n${YELLOW}[5/5] Memeriksa SR-IOV Virtual Functions Terdaftar...${NC}"
TOTAL_VFS=$(find /sys/bus/pci/devices/ -name "sriov_numvfs" 2>/dev/null | xargs cat 2>/dev/null | awk '{s+=$1} END {print s+0}')
echo -e "  - Total VF Aktif: ${BLUE}$TOTAL_VFS${NC}"

echo -e "\n${BLUE}===================================================================${NC}"
echo -e "Ringkasan Audit: Errors: ${RED}$ERRORS${NC} | Warnings: ${YELLOW}$WARNINGS${NC}"
if [ "$ERRORS" -eq 0 ]; then
    echo -e "${GREEN}STATUS: Host memenuhi kriteria operasi DPDK / SR-IOV!${NC}"
else
    echo -e "${RED}STATUS: Kegagalan konfigurasi ditemukan. Tinjau kembali parameter bootloader.${NC}"
fi
echo -e "${BLUE}===================================================================${NC}"

#!/usr/bin/env bash
# ==============================================================================
# Script: benchmark-dpdk-vpp-throughput.sh
# Pengukur Throughput Jaringan Presisi Tinggi & Hardware Drop Counter
# ==============================================================================
set -eo pipefail

IFACE=${1:-""}
DURATION=${2:-10}

if [ -z "$IFACE" ]; then
    echo "Penggunaan: $0 <interface_name> [duration_seconds]"
    exit 1
fi

get_stats() {
    local dev=$1
    local rxb=$(cat /sys/class/net/"$dev"/statistics/rx_bytes 2>/dev/null || echo 0)
    local txb=$(cat /sys/class/net/"$dev"/statistics/tx_bytes 2>/dev/null || echo 0)
    local rxp=$(cat /sys/class/net/"$dev"/statistics/rx_packets 2>/dev/null || echo 0)
    local txp=$(cat /sys/class/net/"$dev"/statistics/tx_packets 2>/dev/null || echo 0)
    local rxd=$(cat /sys/class/net/"$dev"/statistics/rx_dropped 2>/dev/null || echo 0)
    local txd=$(cat /sys/class/net/"$dev"/statistics/tx_dropped 2>/dev/null || echo 0)
    echo "$rxb $txb $rxp $txp $rxd $txd"
}

read -r rxb1 txb1 rxp1 txp1 rxd1 txd1 <<< "$(get_stats "$IFACE")"
START_TIME=$(date +%s.%N)

echo "Memantau trafik interface $IFACE selama $DURATION detik..."
sleep "$DURATION"

read -r rxb2 txb2 rxp2 txp2 rxd2 txd2 <<< "$(get_stats "$IFACE")"
END_TIME=$(date +%s.%N)

ELAPSED=$(awk "BEGIN {print $END_TIME - $START_TIME}")
DELTA_RX_BYTES=$((rxb2 - rxb1))
DELTA_TX_BYTES=$((txb2 - txb1))
DELTA_RX_PKTS=$((rxp2 - rxp1))
DELTA_TX_PKTS=$((txp2 - txp1))
DELTA_RX_DROP=$((rxd2 - rxd1))
DELTA_TX_DROP=$((txd2 - txd1))

RX_GBPS=$(awk "BEGIN {printf \"%.4f\", ($DELTA_RX_BYTES * 8) / ($ELAPSED * 1000000000)}")
TX_GBPS=$(awk "BEGIN {printf \"%.4f\", ($DELTA_TX_BYTES * 8) / ($ELAPSED * 1000000000)}")
RX_MPPS=$(awk "BEGIN {printf \"%.4f\", $DELTA_RX_PKTS / ($ELAPSED * 1000000)}")
TX_MPPS=$(awk "BEGIN {printf \"%.4f\", $DELTA_TX_PKTS / ($ELAPSED * 1000000)}")

echo "------------------- HASIL BENCHMARK LINE-RATE ---------------------"
echo "Durasi Pengujian : $ELAPSED detik"
echo "Throughput RX    : $RX_GBPS Gbps ($RX_MPPS Mpps)"
echo "Throughput TX    : $TX_GBPS Gbps ($TX_MPPS Mpps)"
echo "Total Paket RX   : $DELTA_RX_PKTS paket"
echo "Total Paket TX   : $DELTA_TX_PKTS paket"
echo "Hardware Drop RX : $DELTA_RX_DROP paket"
echo "Hardware Drop TX : $DELTA_TX_DROP paket"
echo "-------------------------------------------------------------------"