Enabling Native Sidecar Pattern with Istio on GKE

Summary

This document details how we enabled the native sidecar container pattern in a GKE cluster using Anthos Service Mesh (ASM). We describe the limitations we encountered with Google’s managed ASM, the rationale behind switching to a self-managed Istio control plane, and the platform-level adjustments required for integration within Formance infrastructure.

Background

When adopting the native sidecar pattern in Istio, it is possible you encounter architectural constraints imposed by Google Cloud’s managed Cloud Service Mesh (CSM). Specifically, it is not possible to configure sidecar behavior at the operator level when using managed control planes, which prevent you from deploying Jobs and StatefulSets that terminate correctly after running their logic.

These limitations are documented in Google’s Service Mesh onboarding documentation, and align with our observed behavior.

Problem Summary

Provisioning Cloud Service Mesh via GCP’s UI or asmcli in managed mode will prevent you from leveraging native sidecars across these Kubernetes resource types:

• ❌ Jobs — can not complete due to long-lived sidecars

• ❌ StatefulSets — sidecars outlive main container lifecycle

• ❌ DaemonSets, Pods with manual injection — unsupported or inconsistently injected

• ❌ istio-proxy ran as a standard container, not initContainer

Approach and Resolution

1. Managed ASM: Workarounds (Not Ideal)

Initial attempts to use managed ASM required complex, brittle workarounds:

• Inject polling containers before primary workloads to delay readiness.

• Manually configure Job successPolicy=NonIndexed and timeouts.

• No access to Istio operator or ENABLE_NATIVE_SIDECARS.

Given that the injection logic is handled by Google’s tenant-managed operator, we couldn’t set the required environment variable or toggle native injection at the cluster level.

2. Switched to Self-Managed ASM

To overcome these restrictions, we moved to self-managed ASM, using asmcli to deploy our own control plane:

./asmcli install \
  --output_dir out \
  --kc $KUBECONFIG \
  --co overlay/native_sidecar.yaml \
  --enable_cluster_labels \
  --enable_cluster_roles

IstioOperator Overlay Example

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    pilot:
      env:
        ENABLE_NATIVE_SIDECARS: "true"

This allowed us to configure the operator environment directly and toggle sidecar behavior cluster-wide.

3. Version Compatibility

We verified that native sidecar support was only available in ASM builds derived from Istio v1.19+.

• Managed ASM defaulted to v1.17 lacked the feature.

• We used 1.19.10-asm.26, provided via internal distribution.

• Referenced GCP release notes – July 2023.

4. Namespace Configuration

After installing the new revision, we updated workloads to target our custom control plane:

metadata:
  labels:
    istio.io/rev: asm-11910-9

5. Outcome

• ✅ istio-proxy is injected as an initContainer, consistent with Kubernetes Sidecar pattern.

• ✅ Job workloads terminate as expected after execution.

• ✅ No polling logic or readiness probes required.

• ✅ StatefulSets and one-shot pods behave consistently.

Platform Integration: Formance

Namespace-Level Labeling

To propagate the Istio revision across all stacks, we introduced a global Settings resource:

apiVersion: formance.com/v1beta1
kind: Settings
metadata:
  name: namespace-label
spec:
  key: namespace.labels
  stacks:
    - "*"
  value: istio.io/rev=asm-11910-9

This ensures fctl automatically labels all Formance workloads for proper injection.

Service Mesh & Console Integration

• ✅ Services are discoverable via formance.cloud console.

• ✅ No manual overrides required in Helm charts.

• ✅ All ingress gateways and stack services are integrated seamlessly.

Observed Challenges and Workarounds

1. Operator Awareness of Injected Sidecars

Our Kubernetes operator couldn’t detect istio-proxy containers, as injection is handled by the MutatingWebhookConfiguration. These containers are invisible to the operator’s reconciliation logic.

✅ Status: Known limitation; will require future enhancement to detect injected initContainers.

2. GCP Traffic Director & NEG Annotations

Early versions of the operator removed necessary annotations required for Google’s Traffic Director:

annotations:
  cloud.google.com/neg: {"ingress":true}

This was fixed in operator v2.11.0, and the settings object now supports annotation passthrough:

apiVersion: formance.com/v1beta1
kind: Settings
metadata:
  name: services-annotations
spec:
  key: services.gateway.annotations
  stacks:
    - "*"
  value: cloud.google.com/neg={"ingress":true,"exposed_ports":{"8080":{}}}

💡 Traffic Director now injects its own annotations. This configuration is primarily useful for static defaults.

Advanced: Why Managed ASM Fails for Native Sidecars

What Is “Managed ASM”?

In Managed ASM, the Istio control plane (including the Istio Operator) is hosted in a Google-managed tenant project, outside your GKE cluster. You cannot modify:

• IstioOperator environment variables

• Injection webhook behavior

• Control plane revisions

Why ENABLE_NATIVE_SIDECARS=true Fails

This variable must be set on the Istio Operator or Istiod, but in managed mode:

• Those components run outside your scope.

• You cannot patch or override their deployment spec.

• Thus, native sidecars cannot be enabled at the source.

What About Pod Annotations?

From Istio 1.23.6+, the following annotation can be used:

sidecar.istio.io/native-sidecar: "true"

This is evaluated by the webhook at injection time — no operator change required.

However, ASM 1.19.10 (which Formance currently uses) does not support the annotation. This renders both options ineffective in managed mode.

Recommendation

To enable native sidecars in GKE with Istio:

• ✅ Use self-managed ASM (with version ≥1.19 and ENABLE_NATIVE_SIDECARS=true)

• OR

• 🔄 Upgrade to ASM version ≥1.23.6 and use the annotation method:

sidecar.istio.io/native-sidecar: "true"

ASM upgrade can be performed via:

./asmcli install \
  --project_id YOUR_PROJECT_ID \
  --cluster_name YOUR_CLUSTER_NAME \
  --cluster_location YOUR_CLUSTER_LOCATION \
  --output_dir ./asm_output \
  --enable_all \
  --channel rapid

Final Notes

• ✅ Native sidecars reduce lifecycle management complexity for short-lived workloads.

• ✅ Jobs, StatefulSets, and ephemeral containers behave predictably.

For additional context, refer to:

• 📘 Istio Native Sidecars Blog

• 📘 GCP Service Mesh Docs

• 📘 Kubernetes Sidecar Pattern