Deployment

Shadow Deployment on Kubernetes with Istio Mirroring

Set up shadow deployment on Kubernetes using Istio traffic mirroring. Covers VirtualService config, monitoring the shadow service and production patterns.

Apoorva Ronad
Apoorva Ronad
5 min read
Shadow Deployment on Kubernetes with Istio Mirroring

Kubernetes with Istio is the most powerful platform for shadow deployments available today. Istio's traffic mirroring feature, built into its VirtualService resource, lets you send a copy of production traffic to any service in your mesh with a single YAML configuration change — no application code modifications, no infrastructure changes, no downtime.

Shadow deployment on Kubernetes with an Istio Envoy sidecar mirroring traffic

Prerequisites

This guide assumes you have a Kubernetes cluster (1.24+), Istio installed with sidecar injection enabled, and a production service already deployed on Kubernetes.

The Istio Traffic Mirroring Architecture

Istio deploys an Envoy proxy as a sidecar alongside every pod. When you configure mirroring in a VirtualService, the Envoy sidecar duplicates every request and sends the copy to the shadow service. The shadow response is discarded at the proxy level — it never propagates back to the caller.

This architecture gives Istio traffic mirroring several important properties: zero latency impact (mirroring is fire-and-forget), independent shadow failures (errors in shadow have zero production effect), and percentage-based mirroring to control load.

Step 1: Deploy the Shadow Service

Your shadow service runs the new version you want to validate. Deploy it as a separate Kubernetes Deployment with a distinct version label.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api-shadow
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-api
      version: shadow
  template:
    metadata:
      labels:
        app: my-api
        version: shadow
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: my-api
        image: my-registry/my-api:v2.0.0
        ports:
        - containerPort: 8080
        env:
        - name: SHADOW_MODE
          value: "true"
        - name: DB_MODE
          value: "readonly"
---
apiVersion: v1
kind: Service
metadata:
  name: my-api-shadow
  namespace: production
spec:
  selector:
    app: my-api
    version: shadow
  ports:
  - port: 80
    targetPort: 8080

The SHADOW_MODE=true environment variable tells the new version to suppress side effects — skip database writes, skip email sending, skip payment charges.

Step 2: Configure Destination Rules

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-api
  namespace: production
spec:
  host: my-api
  subsets:
  - name: production
    labels:
      version: production
  - name: shadow
    labels:
      version: shadow

Step 3: Enable Traffic Mirroring

Shadow deployment traffic mirroring flow

The VirtualService is where you enable mirroring. The mirror field specifies the shadow destination, and mirrorPercentage controls what fraction of traffic is duplicated.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-api
  namespace: production
spec:
  hosts:
  - my-api
  http:
  - route:
    - destination:
        host: my-api
        subset: production
      weight: 100
    mirror:
      host: my-api-shadow
    mirrorPercentage:
      value: 10.0

Apply and verify:

kubectl apply -f shadow-deployment.yaml
kubectl apply -f destination-rule.yaml
kubectl apply -f virtual-service.yaml
kubectl get pods -n production -l version=shadow

Step 4: Gradually Increase Mirror Percentage

Start with 10% mirroring to validate the shadow can handle load, then increase once the shadow is confirmed healthy.

# Increase to 50%
kubectl patch virtualservice my-api -n production \
  --type=json \
  -p='[{"op":"replace","path":"/spec/http/0/mirrorPercentage/value","value":50.0}]'

Monitoring the Shadow Service

Compare shadow behavior against production side-by-side using Istio's automatic metrics.

# Shadow error rate
sum(rate(istio_requests_total{
  destination_service_name="my-api-shadow",
  response_code=~"5.."
}[5m])) /
sum(rate(istio_requests_total{
  destination_service_name="my-api-shadow"
}[5m]))

Build a Grafana dashboard with four panels: P99 latency comparison, error rate comparison, request volume (verify shadow is receiving traffic), and memory usage (detect leaks early).

Handling Writes in Shadow Mode

The hardest part of shadow deployments is preventing side effects from mirrored requests.

import os

def process_order(order):
    if os.getenv("SHADOW_MODE") == "true":
        return simulate_order_processing(order)
    return execute_order(order)

Alternatively, route the shadow service to a read-only database replica or use Istio's EnvoyFilter to intercept and mock outbound calls to downstream services.

Automated Shadow Validation in CI/CD

Integrate shadow validation into your pipeline. After deploying the shadow, run an automated check that queries Prometheus for shadow error rates before promoting to canary.

- name: Enable shadow mirroring
  run: kubectl apply -f k8s/virtual-service-shadow.yaml

- name: Wait for shadow to warm up
  run: sleep 300

- name: Validate shadow error rate below threshold
  run: |
    SHADOW_ERR=$(kubectl exec -n monitoring deploy/prometheus -- \
      promtool query instant http://localhost:9090 \
      'rate(istio_requests_total{destination_service_name="my-api-shadow",response_code=~"5.."}[5m])')
    echo "Shadow error rate: $SHADOW_ERR"

- name: Promote to canary
  run: kubectl apply -f k8s/virtual-service-canary.yaml

Common Issues

Shadow latency much higher than production — Caused by cold database caches. Let the shadow run for 30+ minutes before comparing latency metrics.

Mirrored requests fail with auth errors — The shadow uses the same JWT token as the production request. Ensure the shadow service uses the same authentication secrets and signing keys.

Shadow pods OOMKilled — Start at 10% mirroring to allow the shadow to establish its baseline memory footprint before exposing it to 100% load.

Cleaning Up

Once the new version is validated and promoted to production, remove the shadow.

# Remove mirroring
kubectl patch virtualservice my-api -n production \
  --type=json \
  -p='[{"op":"remove","path":"/spec/http/0/mirror"}]'

kubectl delete deployment my-api-shadow -n production
kubectl delete service my-api-shadow -n production

Conclusion

Istio's traffic mirroring makes shadow deployment on Kubernetes a first-class, operationally lightweight technique. With three YAML files and zero application code changes, you can test your new version against real production traffic before a single user ever sees it. Combining shadow validation with a subsequent canary rollout creates one of the most robust deployment pipelines available.

Related: Shadow Deployment Concepts | Ramped Deployment Strategy

From the team at Nife

Deploy your next app in minutes, not days.

Nife is a multi-cloud edge platform that lets you deploy, scale, and manage applications globally without the infrastructure headache.