Multi-Platform Service Mesh: Connecting Kubernetes, Nomad, Bare Metal, and VMs with Consul Connect

thumbnail
Martin Buchleitner | 02.10.2025 cloud native, hashicorp, devops

Bicycle

Multi-Platform Service Mesh: Connecting Kubernetes, Nomad, Bare Metal, and VMs with Consul Connect

Modern infrastructure often spans multiple platforms - cloud-native Kubernetes clusters, HashiCorp Nomad for orchestration, bare metal servers for performance-critical workloads, and traditional virtual machines for legacy applications. Managing secure communication across these diverse environments can be complex and fragmented.

In this comprehensive guide, we'll demonstrate how to build a unified service mesh that seamlessly connects services across Kubernetes, Nomad, bare metal servers, and VMs using Consul Connect federation. This approach provides consistent security, service discovery, and traffic management regardless of where your services are deployed.

Prerequisites: This article builds upon concepts from Secure Communication in Kubernetes with Consul Connect and Vault Agent Injector. Familiarity with Consul, service mesh concepts, and basic Kubernetes/Nomad operations is recommended.

Architecture Overview

Our multi-platform service mesh creates a unified infrastructure that seamlessly connects services across different deployment environments:

Multi-Platform Consul Connect Architecture

The architecture demonstrates how Consul Connect federation enables secure communication between:

  • Kubernetes DC1: Cloud-native applications with automated sidecar injection and service mesh integration
  • Nomad DC2: Container orchestration with flexible job scheduling and Consul Connect integration
  • Bare Metal/VM DC3: Traditional infrastructure with manual Envoy configuration and legacy application support

All platforms are unified through Consul Federation which provides:

  • Mesh Gateways: Secure cross-datacenter communication
  • mTLS + ACLs: Consistent security policies across all environments
  • WAN Federation: Service discovery and routing across platform boundaries

Benefits of Multi-Platform Service Mesh

1. Unified Security Model

  • Consistent mTLS encryption across all platforms
  • Centralized certificate management
  • Uniform access control policies (intentions)
  • Single identity and trust model

2. Simplified Service Discovery

  • Services can discover each other regardless of platform
  • Consistent DNS-based service resolution
  • Automatic health checking and failover
  • Load balancing across platforms

3. Traffic Management

  • Traffic routing policies that work across environments
  • Circuit breaking and retry logic
  • Blue-green deployments spanning platforms
  • Canary releases with cross-platform traffic splitting

4. Operational Consistency

  • Single pane of glass for service mesh operations
  • Consistent logging and metrics collection
  • Unified troubleshooting and debugging
  • Centralized configuration management

Step 1: Consul Federation Setup

First, let's establish Consul federation to connect our different datacenters.

Primary Datacenter (Kubernetes) Setup

 1# consul-dc1-values.yaml - Primary datacenter in Kubernetes
 2global:
 3  name: consul
 4  image: "hashicorp/consul:1.16.3"
 5  datacenter: dc1
 6  primaryDatacenter: dc1
 7  peering:
 8    enabled: true
 9  # Enable federation
10  federation:
11    enabled: true
12  # Enable mesh gateways for cross-datacenter communication
13  meshGateway:
14    enabled: true
15    replicas: 2
16  # Enable ACLs for security
17  acls:
18    manageSystemACLs: true
19    createReplicationToken: true
20
21server:
22  enabled: true
23  replicas: 3
24  bootstrapExpect: 3
25  # Enable Connect for service mesh
26  connect: true
27  # Expose federation state endpoint
28  exposeGossipAndRPCPorts: true
29  ports:
30    # Federation endpoint
31    serfWAN: 8302
32
33connectInject:
34  enabled: true
35  default: false
36  # Enable transparent proxy for easier service communication
37  transparentProxy:
38    defaultEnabled: true
39  # Consul Connect injection settings
40  consulNode:
41    meta:
42      pod-name: \${HOSTNAME}
43      node-name: \${NODE_NAME}
44  k8sAllowNamespaces: ["*"]
45  k8sDenyNamespaces: ["kube-system", "consul"]
46
47meshGateway:
48  enabled: true
49  replicas: 2
50  service:
51    type: LoadBalancer
52    ports:
53      - port: 443
54        nodePort: null
55  # Enable WAN federation through mesh gateways
56  wanAddress:
57    source: "Service"
58    port: 443
59
60ui:
61  enabled: true
62  service:
63    type: LoadBalancer

Deploy the primary Consul datacenter:

 1# Create consul namespace
 2kubectl create namespace consul
 3
 4# Add Consul Helm repository
 5helm repo add hashicorp https://helm.releases.hashicorp.com
 6helm repo update
 7
 8# Install primary Consul datacenter
 9helm install consul-dc1 hashicorp/consul \
10  --namespace consul \
11  --values consul-dc1-values.yaml
12
13# Wait for deployment
14kubectl wait --for=condition=ready pod -l app=consul -n consul --timeout=300s
15
16# Get the federation secret for secondary datacenters
17kubectl get secret consul-dc1-federation -n consul -o yaml > consul-federation-secret.yaml

Secondary Datacenter (Nomad) Setup

Create Consul configuration for Nomad integration:

 1# consul-dc2.hcl - Secondary datacenter for Nomad integration
 2datacenter = "dc2"
 3primary_datacenter = "dc1"
 4log_level = "INFO"
 5node_name = "consul-dc2"
 6bind_addr = "0.0.0.0"
 7client_addr = "0.0.0.0"
 8
 9# Server configuration
10server = true
11bootstrap_expect = 1
12ui_config {
13  enabled = true
14}
15
16# Enable Connect for service mesh
17connect {
18  enabled = true
19}
20
21# Enable mesh gateway
22ports {
23  grpc = 8502
24  mesh_gateway = 8443
25}
26
27# Federation configuration
28retry_join_wan = ["KUBERNETES_MESH_GATEWAY_ADDRESS:443"]
29
30# ACL configuration
31acl = {
32  enabled = true
33  default_policy = "deny"
34  enable_token_persistence = true
35  tokens {
36    replication = "REPLICATION_TOKEN_FROM_PRIMARY"
37  }
38}
39
40# Auto-encrypt configuration
41auto_encrypt {
42  allow_tls = true
43}

Nomad Configuration with Consul Connect

 1# nomad-dc2.hcl - Nomad configuration for service mesh integration
 2datacenter = "dc2"
 3name = "nomad-dc2"
 4region = "global"
 5
 6bind_addr = "0.0.0.0"
 7
 8# Server configuration
 9server {
10  enabled = true
11  bootstrap_expect = 1
12}
13
14# Client configuration
15client {
16  enabled = true
17  # Enable task networking for Connect
18  cni_path = "/opt/cni/bin"
19  cni_config_dir = "/opt/cni/config"
20}
21
22# Consul integration
23consul {
24  address = "127.0.0.1:8500"
25  server_service_name = "nomad"
26  client_service_name = "nomad-client"
27  auto_advertise = true
28  server_auto_join = true
29  client_auto_join = true
30
31  # Enable service mesh integration
32  connect {
33    enabled = true
34  }
35}
36
37# Plugin configuration for networking
38plugin "docker" {
39  config {
40    allow_privileged = true
41    volumes {
42      enabled = true
43    }
44  }
45}

Step 2: Cross-Platform Service Deployment

Kubernetes Web Application

 1# web-app-k8s.yaml - Web application in Kubernetes
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5  name: web-app
 6  namespace: demo
 7spec:
 8  replicas: 2
 9  selector:
10    matchLabels:
11      app: web-app
12  template:
13    metadata:
14      labels:
15        app: web-app
16      annotations:
17        "consul.hashicorp.com/connect-inject": "true"
18        "consul.hashicorp.com/connect-service": "web-app"
19        "consul.hashicorp.com/connect-service-upstreams": "api-service:8080:dc2,database:5432:dc3"
20        "consul.hashicorp.com/transparent-proxy": "true"
21    spec:
22      serviceAccountName: web-app
23      containers:
24      - name: web-app
25        image: nginx:latest
26        ports:
27        - containerPort: 80
28        env:
29        - name: API_SERVICE_URL
30          value: "http://api-service:8080"
31        - name: DATABASE_URL
32          value: "postgres://database:5432/mydb"
33
34---
35apiVersion: v1
36kind: Service
37metadata:
38  name: web-app
39  namespace: demo
40spec:
41  selector:
42    app: web-app
43  ports:
44  - port: 80
45    targetPort: 80
46  type: LoadBalancer

Nomad API Service

 1# api-service-nomad.nomad - API service in Nomad
 2job "api-service" {
 3  datacenters = ["dc2"]
 4  type = "service"
 5
 6  group "api" {
 7    count = 2
 8
 9    # Enable Consul Connect
10    service {
11      name = "api-service"
12      port = "http"
13
14      connect {
15        sidecar_service {
16          proxy {
17            upstreams {
18              destination_name = "database"
19              datacenter = "dc3"
20              local_bind_port = 5432
21            }
22          }
23        }
24      }
25    }
26
27    network {
28      mode = "bridge"
29      port "http" {
30        static = 8080
31        to = 8080
32      }
33    }
34
35    task "api-server" {
36      driver = "docker"
37
38      config {
39        image = "hashicorp/http-echo:latest"
40        args = [
41          "-listen", ":8080",
42          "-text", "API Service from Nomad DC2"
43        ]
44      }
45
46      env {
47        DATABASE_URL = "postgres://127.0.0.1:5432/mydb"
48      }
49
50      resources {
51        cpu = 256
52        memory = 256
53      }
54    }
55  }
56}

Deploy the Nomad job:

1# Deploy API service to Nomad
2nomad job run api-service-nomad.nomad
3
4# Verify deployment
5nomad job status api-service
6consul services register api-service

Bare Metal Database Service

 1# install-consul-bare-metal.sh - Install Consul on bare metal
 2#!/bin/bash
 3
 4# Download and install Consul
 5CONSUL_VERSION="1.16.3"
 6wget https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip
 7unzip consul_${CONSUL_VERSION}_linux_amd64.zip
 8sudo mv consul /usr/local/bin/
 9
10# Create consul user and directories
11sudo useradd --system --home /etc/consul.d --shell /bin/false consul
12sudo mkdir -p /opt/consul /etc/consul.d
13sudo chown consul:consul /opt/consul /etc/consul.d
14sudo chmod 755 /opt/consul /etc/consul.d

 1# consul-dc3.hcl - Consul configuration for bare metal
 2datacenter = "dc3"
 3primary_datacenter = "dc1"
 4log_level = "INFO"
 5node_name = "consul-dc3-server"
 6bind_addr = "192.168.1.100"  # Replace with actual IP
 7client_addr = "0.0.0.0"
 8data_dir = "/opt/consul"
 9
10# Server configuration
11server = true
12bootstrap_expect = 1
13
14# Connect configuration
15connect {
16  enabled = true
17}
18
19ports {
20  grpc = 8502
21  mesh_gateway = 8443
22}
23
24# Join the federation
25retry_join_wan = ["KUBERNETES_MESH_GATEWAY_ADDRESS:443"]
26
27# ACL configuration
28acl = {
29  enabled = true
30  default_policy = "deny"
31  enable_token_persistence = true
32  tokens {
33    replication = "REPLICATION_TOKEN_FROM_PRIMARY"
34    agent = "AGENT_TOKEN"
35  }
36}
37
38# Service definitions
39services {
40  name = "database"
41  id = "postgresql-primary"
42  address = "192.168.1.100"
43  port = 5432
44
45  check {
46    tcp = "192.168.1.100:5432"
47    interval = "10s"
48  }
49
50  connect {
51    sidecar_service {
52      port = 21000
53      check {
54        name = "Connect Envoy Sidecar"
55        tcp = "192.168.1.100:21000"
56        interval = "10s"
57      }
58      proxy {
59        config {
60          bind_address = "192.168.1.100"
61          bind_port = 21000
62        }
63      }
64    }
65  }
66}

VM Service Registration

 1# register-vm-service.sh - Register VM-based services
 2#!/bin/bash
 3
 4# Install Consul agent on VM
 5curl -fsSL https://releases.hashicorp.com/consul/1.16.3/consul_1.16.3_linux_amd64.zip -o consul.zip
 6unzip consul.zip
 7sudo mv consul /usr/local/bin/
 8
 9# Create service definition
10cat > /etc/consul.d/vm-service.json <<EOF
11{
12  "service": {
13    "name": "legacy-app",
14    "id": "legacy-app-vm1",
15    "address": "192.168.1.101",
16    "port": 9090,
17    "tags": ["vm", "legacy"],
18    "check": {
19      "http": "http://192.168.1.101:9090/health",
20      "interval": "30s"
21    },
22    "connect": {
23      "sidecar_service": {
24        "port": 21001,
25        "check": {
26          "name": "Connect Envoy Sidecar",
27          "tcp": "192.168.1.101:21001",
28          "interval": "10s"
29        }
30      }
31    }
32  }
33}
34EOF
35
36# Start Consul agent
37consul agent -config-dir=/etc/consul.d -datacenter=dc3 \
38  -retry-join=192.168.1.100 -bind=192.168.1.101

Step 3: Service Intentions and Security

Configure service intentions for cross-platform security:

 1# Create service intentions for cross-datacenter communication
 2
 3# Allow web-app (K8s DC1) to communicate with api-service (Nomad DC2)
 4consul intention create -allow web-app api-service
 5
 6# Allow api-service (Nomad DC2) to communicate with database (Bare Metal DC3)
 7consul intention create -allow api-service database
 8
 9# Allow api-service to communicate with legacy-app (VM DC3)
10consul intention create -allow api-service legacy-app
11
12# Verify intentions
13consul intention list

Create intention configuration files for GitOps:

 1# service-intentions.yaml
 2apiVersion: consul.hashicorp.com/v1alpha1
 3kind: ServiceIntentions
 4metadata:
 5  name: api-service-intentions
 6  namespace: consul
 7spec:
 8  destination:
 9    name: api-service
10  sources:
11    - name: web-app
12      action: allow
13      description: "Allow web-app to call api-service"
14    - name: "*"
15      action: deny
16      description: "Deny all other traffic"
17
18---
19apiVersion: consul.hashicorp.com/v1alpha1
20kind: ServiceIntentions
21metadata:
22  name: database-intentions
23  namespace: consul
24spec:
25  destination:
26    name: database
27  sources:
28    - name: api-service
29      action: allow
30      description: "Allow api-service to access database"
31    - name: "*"
32      action: deny
33      description: "Deny all other traffic"

Step 4: Mesh Gateway Configuration

Configure mesh gateways for secure cross-datacenter communication:

 1# mesh-gateway-config.yaml
 2apiVersion: consul.hashicorp.com/v1alpha1
 3kind: ProxyDefaults
 4metadata:
 5  name: global
 6  namespace: consul
 7spec:
 8  meshGateway:
 9    mode: "local"
10  config:
11    protocol: "http"
12
13---
14apiVersion: consul.hashicorp.com/v1alpha1
15kind: ServiceDefaults
16metadata:
17  name: api-service
18  namespace: consul
19spec:
20  protocol: "http"
21  meshGateway:
22    mode: "local"
23
24---
25apiVersion: consul.hashicorp.com/v1alpha1
26kind: ServiceDefaults
27metadata:
28  name: database
29  namespace: consul
30spec:
31  protocol: "tcp"
32  meshGateway:
33    mode: "local"

Apply mesh gateway configurations:

1kubectl apply -f mesh-gateway-config.yaml
2kubectl apply -f service-intentions.yaml

Step 5: Health Checking and Load Balancing

Implement comprehensive health checking across platforms:

 1# health-check-config.yaml
 2apiVersion: consul.hashicorp.com/v1alpha1
 3kind: ServiceDefaults
 4metadata:
 5  name: api-service
 6  namespace: consul
 7spec:
 8  protocol: "http"
 9  upstreamConfig:
10    defaults:
11      connectTimeoutMs: 5000
12      limits:
13        maxConnections: 50
14        maxPendingRequests: 100
15      passiveHealthCheck:
16        interval: "30s"
17        maxFailures: 3

Configure load balancing policies:

 1# load-balancer-config.hcl
 2Kind = "service-resolver"
 3Name = "api-service"
 4
 5LoadBalancer = {
 6  Policy = "round_robin"
 7  RingHashConfig = {
 8    MinimumRingSize = 1024
 9    MaximumRingSize = 8192
10  }
11  HashPolicies = [
12    {
13      Field = "header"
14      FieldValue = "x-user-id"
15    }
16  ]
17}
18
19Failover = {
20  "*" = {
21    Datacenters = ["dc1", "dc3"]
22  }
23}

Step 6: Traffic Management and Routing

Implement advanced traffic management across platforms:

 1# traffic-splitting.yaml
 2apiVersion: consul.hashicorp.com/v1alpha1
 3kind: ServiceSplitter
 4metadata:
 5  name: api-service
 6  namespace: consul
 7spec:
 8  splits:
 9    - weight: 80
10      service: api-service
11      serviceSubset: v1
12    - weight: 20
13      service: api-service
14      serviceSubset: v2
15
16---
17apiVersion: consul.hashicorp.com/v1alpha1
18kind: ServiceResolver
19metadata:
20  name: api-service
21  namespace: consul
22spec:
23  defaultSubset: v1
24  subsets:
25    v1:
26      filter: "Service.Tags contains v1"
27    v2:
28      filter: "Service.Tags contains v2"
29  failover:
30    "*":
31      datacenters: ["dc1", "dc3"]

Best Practices

1. Network Segmentation

  • Use separate network segments for each platform
  • Configure firewall rules for mesh gateway communication
  • Implement network policies in Kubernetes

2. Security Hardening

  • Enable ACLs in all datacenters
  • Use service intentions for fine-grained access control
  • Rotate certificates regularly

3. Operational Excellence

  • Deploy consistent logging and metrics collection
  • Set up alerts for mesh gateway health
  • Monitor certificate expiration across all platforms

4. Deployment Automation

  • Use GitOps for intention and configuration management
  • Automate service registration across platforms
  • Implement blue-green deployments spanning multiple environments

Conclusion

Building a multi-platform service mesh with Consul Connect enables organizations to:

  • Unify Security: Consistent mTLS encryption and access control across all platforms
  • Simplify Operations: Single service mesh spanning diverse infrastructure
  • Enable Innovation: Deploy services on the most appropriate platform without losing connectivity
  • Reduce Complexity: One service discovery and communication model for all environments
  • Improve Reliability: Cross-platform failover and load balancing

This architecture provides the foundation for modern hybrid and multi-cloud deployments, allowing teams to leverage the best features of each platform while maintaining security and operational consistency.

Key takeaways:

  • Federation First: Establish Consul federation as the foundation for cross-platform communication
  • Security by Design: Use service intentions and mTLS for zero-trust networking
  • Platform Flexibility: Deploy services where they make the most sense operationally
  • Consistent Operations: Same tools and processes across all platforms
  • Gradual Migration: Incrementally move services between platforms without breaking connectivity

Ready to build your multi-platform service mesh? Start with establishing Consul federation between your most critical environments and gradually expand coverage to achieve complete platform unification.

For comprehensive monitoring and observability of your multi-platform service mesh, see our companion article on Advanced Monitoring and Observability for Consul Connect Service Mesh.

Go Back explore our courses
Schedule a quick initial call
Similar Articles & Courses
Our Recommendation

Cloud Native Essentials: Kubernetes

This is an introduction to Kubernetes, an open-source system for automating deployment, scaling, and management of containerized applications.

Vault Enterprise

Learn how to use HashiCorp Vault Enterprise to centrally secure your sensitive data and meet enterprise-grade compliance and operations requirements.

Consul Enterprise

Connect dynamic applications.

thumbnail
Matthias Theuermann | 05.11.2025 security, hashicorp, devops

Keeping Credentials Out of Code - A Practical Guide to 1Password and Vault

The Problem: Hardcoded Credentials Every developer has faced this temptation: you need to test something quickly, so you hardcode an API key or database

thumbnail
Jürgen Brüder | 21.10.2025 devops, cloud native

Docker vs Podman - Choosing the Right Container Platform for Your Team

The container ecosystem has evolved significantly over the past few years, and teams today have more choices than ever when selecting their container runtime

We are here for you

You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.

Contact us