Shadow Deployment on Kubernetes with Istio Mirroring
Set up shadow deployment on Kubernetes using Istio traffic mirroring. Covers VirtualService config, monitoring the shadow service and production patterns.
Kubernetes with Istio is the most powerful platform for shadow deployments available today. Istio's traffic mirroring feature, built into its VirtualService resource, lets you send a copy of production traffic to any service in your mesh with a single YAML configuration change — no application code modifications, no infrastructure changes, no downtime.
Prerequisites
This guide assumes you have a Kubernetes cluster (1.24+), Istio installed with sidecar injection enabled, and a production service already deployed on Kubernetes.
The Istio Traffic Mirroring Architecture
Istio deploys an Envoy proxy as a sidecar alongside every pod. When you configure mirroring in a VirtualService, the Envoy sidecar duplicates every request and sends the copy to the shadow service. The shadow response is discarded at the proxy level — it never propagates back to the caller.
This architecture gives Istio traffic mirroring several important properties: zero latency impact (mirroring is fire-and-forget), independent shadow failures (errors in shadow have zero production effect), and percentage-based mirroring to control load.
Step 1: Deploy the Shadow Service
Your shadow service runs the new version you want to validate. Deploy it as a separate Kubernetes Deployment with a distinct version label.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-api-shadow
namespace: production
spec:
replicas: 2
selector:
matchLabels:
app: my-api
version: shadow
template:
metadata:
labels:
app: my-api
version: shadow
annotations:
sidecar.istio.io/inject: "true"
spec:
containers:
- name: my-api
image: my-registry/my-api:v2.0.0
ports:
- containerPort: 8080
env:
- name: SHADOW_MODE
value: "true"
- name: DB_MODE
value: "readonly"
---
apiVersion: v1
kind: Service
metadata:
name: my-api-shadow
namespace: production
spec:
selector:
app: my-api
version: shadow
ports:
- port: 80
targetPort: 8080
The SHADOW_MODE=true environment variable tells the new version to suppress side effects — skip database writes, skip email sending, skip payment charges.
Step 2: Configure Destination Rules
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: my-api
namespace: production
spec:
host: my-api
subsets:
- name: production
labels:
version: production
- name: shadow
labels:
version: shadow
Step 3: Enable Traffic Mirroring
The VirtualService is where you enable mirroring. The mirror field specifies the shadow destination, and mirrorPercentage controls what fraction of traffic is duplicated.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-api
namespace: production
spec:
hosts:
- my-api
http:
- route:
- destination:
host: my-api
subset: production
weight: 100
mirror:
host: my-api-shadow
mirrorPercentage:
value: 10.0
Apply and verify:
kubectl apply -f shadow-deployment.yaml
kubectl apply -f destination-rule.yaml
kubectl apply -f virtual-service.yaml
kubectl get pods -n production -l version=shadow
Step 4: Gradually Increase Mirror Percentage
Start with 10% mirroring to validate the shadow can handle load, then increase once the shadow is confirmed healthy.
# Increase to 50%
kubectl patch virtualservice my-api -n production \
--type=json \
-p='[{"op":"replace","path":"/spec/http/0/mirrorPercentage/value","value":50.0}]'
Monitoring the Shadow Service
Compare shadow behavior against production side-by-side using Istio's automatic metrics.
# Shadow error rate
sum(rate(istio_requests_total{
destination_service_name="my-api-shadow",
response_code=~"5.."
}[5m])) /
sum(rate(istio_requests_total{
destination_service_name="my-api-shadow"
}[5m]))
Build a Grafana dashboard with four panels: P99 latency comparison, error rate comparison, request volume (verify shadow is receiving traffic), and memory usage (detect leaks early).
Handling Writes in Shadow Mode
The hardest part of shadow deployments is preventing side effects from mirrored requests.
import os
def process_order(order):
if os.getenv("SHADOW_MODE") == "true":
return simulate_order_processing(order)
return execute_order(order)
Alternatively, route the shadow service to a read-only database replica or use Istio's EnvoyFilter to intercept and mock outbound calls to downstream services.
Automated Shadow Validation in CI/CD
Integrate shadow validation into your pipeline. After deploying the shadow, run an automated check that queries Prometheus for shadow error rates before promoting to canary.
- name: Enable shadow mirroring
run: kubectl apply -f k8s/virtual-service-shadow.yaml
- name: Wait for shadow to warm up
run: sleep 300
- name: Validate shadow error rate below threshold
run: |
SHADOW_ERR=$(kubectl exec -n monitoring deploy/prometheus -- \
promtool query instant http://localhost:9090 \
'rate(istio_requests_total{destination_service_name="my-api-shadow",response_code=~"5.."}[5m])')
echo "Shadow error rate: $SHADOW_ERR"
- name: Promote to canary
run: kubectl apply -f k8s/virtual-service-canary.yaml
Common Issues
Shadow latency much higher than production — Caused by cold database caches. Let the shadow run for 30+ minutes before comparing latency metrics.
Mirrored requests fail with auth errors — The shadow uses the same JWT token as the production request. Ensure the shadow service uses the same authentication secrets and signing keys.
Shadow pods OOMKilled — Start at 10% mirroring to allow the shadow to establish its baseline memory footprint before exposing it to 100% load.
Cleaning Up
Once the new version is validated and promoted to production, remove the shadow.
# Remove mirroring
kubectl patch virtualservice my-api -n production \
--type=json \
-p='[{"op":"remove","path":"/spec/http/0/mirror"}]'
kubectl delete deployment my-api-shadow -n production
kubectl delete service my-api-shadow -n production
Conclusion
Istio's traffic mirroring makes shadow deployment on Kubernetes a first-class, operationally lightweight technique. With three YAML files and zero application code changes, you can test your new version against real production traffic before a single user ever sees it. Combining shadow validation with a subsequent canary rollout creates one of the most robust deployment pipelines available.
Related: Shadow Deployment Concepts | Ramped Deployment Strategy
From the team at Nife
Deploy your next app in minutes, not days.
Nife is a multi-cloud edge platform that lets you deploy, scale, and manage applications globally without the infrastructure headache.