Modern infrastructure often spans multiple platforms - cloud-native Kubernetes clusters, HashiCorp Nomad for orchestration, bare metal servers for performance-critical workloads, and traditional virtual machines for legacy applications. Managing secure communication across these diverse environments can be complex and fragmented.
In this comprehensive guide, we'll demonstrate how to build a unified service mesh that seamlessly connects services across Kubernetes, Nomad, bare metal servers, and VMs using Consul Connect federation. This approach provides consistent security, service discovery, and traffic management regardless of where your services are deployed.
Prerequisites: This article builds upon concepts from Secure Communication in Kubernetes with Consul Connect and Vault Agent Injector. Familiarity with Consul, service mesh concepts, and basic Kubernetes/Nomad operations is recommended.
Our multi-platform service mesh creates a unified infrastructure that seamlessly connects services across different deployment environments:
The architecture demonstrates how Consul Connect federation enables secure communication between:
All platforms are unified through Consul Federation which provides:
First, let's establish Consul federation to connect our different datacenters.
1# consul-dc1-values.yaml - Primary datacenter in Kubernetes
2global:
3 name: consul
4 image: "hashicorp/consul:1.16.3"
5 datacenter: dc1
6 primaryDatacenter: dc1
7 peering:
8 enabled: true
9 # Enable federation
10 federation:
11 enabled: true
12 # Enable mesh gateways for cross-datacenter communication
13 meshGateway:
14 enabled: true
15 replicas: 2
16 # Enable ACLs for security
17 acls:
18 manageSystemACLs: true
19 createReplicationToken: true
20
21server:
22 enabled: true
23 replicas: 3
24 bootstrapExpect: 3
25 # Enable Connect for service mesh
26 connect: true
27 # Expose federation state endpoint
28 exposeGossipAndRPCPorts: true
29 ports:
30 # Federation endpoint
31 serfWAN: 8302
32
33connectInject:
34 enabled: true
35 default: false
36 # Enable transparent proxy for easier service communication
37 transparentProxy:
38 defaultEnabled: true
39 # Consul Connect injection settings
40 consulNode:
41 meta:
42 pod-name: \${HOSTNAME}
43 node-name: \${NODE_NAME}
44 k8sAllowNamespaces: ["*"]
45 k8sDenyNamespaces: ["kube-system", "consul"]
46
47meshGateway:
48 enabled: true
49 replicas: 2
50 service:
51 type: LoadBalancer
52 ports:
53 - port: 443
54 nodePort: null
55 # Enable WAN federation through mesh gateways
56 wanAddress:
57 source: "Service"
58 port: 443
59
60ui:
61 enabled: true
62 service:
63 type: LoadBalancer
Deploy the primary Consul datacenter:
1# Create consul namespace
2kubectl create namespace consul
3
4# Add Consul Helm repository
5helm repo add hashicorp https://helm.releases.hashicorp.com
6helm repo update
7
8# Install primary Consul datacenter
9helm install consul-dc1 hashicorp/consul \
10 --namespace consul \
11 --values consul-dc1-values.yaml
12
13# Wait for deployment
14kubectl wait --for=condition=ready pod -l app=consul -n consul --timeout=300s
15
16# Get the federation secret for secondary datacenters
17kubectl get secret consul-dc1-federation -n consul -o yaml > consul-federation-secret.yaml
Create Consul configuration for Nomad integration:
1# consul-dc2.hcl - Secondary datacenter for Nomad integration
2datacenter = "dc2"
3primary_datacenter = "dc1"
4log_level = "INFO"
5node_name = "consul-dc2"
6bind_addr = "0.0.0.0"
7client_addr = "0.0.0.0"
8
9# Server configuration
10server = true
11bootstrap_expect = 1
12ui_config {
13 enabled = true
14}
15
16# Enable Connect for service mesh
17connect {
18 enabled = true
19}
20
21# Enable mesh gateway
22ports {
23 grpc = 8502
24 mesh_gateway = 8443
25}
26
27# Federation configuration
28retry_join_wan = ["KUBERNETES_MESH_GATEWAY_ADDRESS:443"]
29
30# ACL configuration
31acl = {
32 enabled = true
33 default_policy = "deny"
34 enable_token_persistence = true
35 tokens {
36 replication = "REPLICATION_TOKEN_FROM_PRIMARY"
37 }
38}
39
40# Auto-encrypt configuration
41auto_encrypt {
42 allow_tls = true
43}
1# nomad-dc2.hcl - Nomad configuration for service mesh integration
2datacenter = "dc2"
3name = "nomad-dc2"
4region = "global"
5
6bind_addr = "0.0.0.0"
7
8# Server configuration
9server {
10 enabled = true
11 bootstrap_expect = 1
12}
13
14# Client configuration
15client {
16 enabled = true
17 # Enable task networking for Connect
18 cni_path = "/opt/cni/bin"
19 cni_config_dir = "/opt/cni/config"
20}
21
22# Consul integration
23consul {
24 address = "127.0.0.1:8500"
25 server_service_name = "nomad"
26 client_service_name = "nomad-client"
27 auto_advertise = true
28 server_auto_join = true
29 client_auto_join = true
30
31 # Enable service mesh integration
32 connect {
33 enabled = true
34 }
35}
36
37# Plugin configuration for networking
38plugin "docker" {
39 config {
40 allow_privileged = true
41 volumes {
42 enabled = true
43 }
44 }
45}
1# web-app-k8s.yaml - Web application in Kubernetes
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: web-app
6 namespace: demo
7spec:
8 replicas: 2
9 selector:
10 matchLabels:
11 app: web-app
12 template:
13 metadata:
14 labels:
15 app: web-app
16 annotations:
17 "consul.hashicorp.com/connect-inject": "true"
18 "consul.hashicorp.com/connect-service": "web-app"
19 "consul.hashicorp.com/connect-service-upstreams": "api-service:8080:dc2,database:5432:dc3"
20 "consul.hashicorp.com/transparent-proxy": "true"
21 spec:
22 serviceAccountName: web-app
23 containers:
24 - name: web-app
25 image: nginx:latest
26 ports:
27 - containerPort: 80
28 env:
29 - name: API_SERVICE_URL
30 value: "http://api-service:8080"
31 - name: DATABASE_URL
32 value: "postgres://database:5432/mydb"
33
34---
35apiVersion: v1
36kind: Service
37metadata:
38 name: web-app
39 namespace: demo
40spec:
41 selector:
42 app: web-app
43 ports:
44 - port: 80
45 targetPort: 80
46 type: LoadBalancer
1# api-service-nomad.nomad - API service in Nomad
2job "api-service" {
3 datacenters = ["dc2"]
4 type = "service"
5
6 group "api" {
7 count = 2
8
9 # Enable Consul Connect
10 service {
11 name = "api-service"
12 port = "http"
13
14 connect {
15 sidecar_service {
16 proxy {
17 upstreams {
18 destination_name = "database"
19 datacenter = "dc3"
20 local_bind_port = 5432
21 }
22 }
23 }
24 }
25 }
26
27 network {
28 mode = "bridge"
29 port "http" {
30 static = 8080
31 to = 8080
32 }
33 }
34
35 task "api-server" {
36 driver = "docker"
37
38 config {
39 image = "hashicorp/http-echo:latest"
40 args = [
41 "-listen", ":8080",
42 "-text", "API Service from Nomad DC2"
43 ]
44 }
45
46 env {
47 DATABASE_URL = "postgres://127.0.0.1:5432/mydb"
48 }
49
50 resources {
51 cpu = 256
52 memory = 256
53 }
54 }
55 }
56}
Deploy the Nomad job:
1# Deploy API service to Nomad
2nomad job run api-service-nomad.nomad
3
4# Verify deployment
5nomad job status api-service
6consul services register api-service
1# install-consul-bare-metal.sh - Install Consul on bare metal
2#!/bin/bash
3
4# Download and install Consul
5CONSUL_VERSION="1.16.3"
6wget https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip
7unzip consul_${CONSUL_VERSION}_linux_amd64.zip
8sudo mv consul /usr/local/bin/
9
10# Create consul user and directories
11sudo useradd --system --home /etc/consul.d --shell /bin/false consul
12sudo mkdir -p /opt/consul /etc/consul.d
13sudo chown consul:consul /opt/consul /etc/consul.d
14sudo chmod 755 /opt/consul /etc/consul.d
1# consul-dc3.hcl - Consul configuration for bare metal
2datacenter = "dc3"
3primary_datacenter = "dc1"
4log_level = "INFO"
5node_name = "consul-dc3-server"
6bind_addr = "192.168.1.100" # Replace with actual IP
7client_addr = "0.0.0.0"
8data_dir = "/opt/consul"
9
10# Server configuration
11server = true
12bootstrap_expect = 1
13
14# Connect configuration
15connect {
16 enabled = true
17}
18
19ports {
20 grpc = 8502
21 mesh_gateway = 8443
22}
23
24# Join the federation
25retry_join_wan = ["KUBERNETES_MESH_GATEWAY_ADDRESS:443"]
26
27# ACL configuration
28acl = {
29 enabled = true
30 default_policy = "deny"
31 enable_token_persistence = true
32 tokens {
33 replication = "REPLICATION_TOKEN_FROM_PRIMARY"
34 agent = "AGENT_TOKEN"
35 }
36}
37
38# Service definitions
39services {
40 name = "database"
41 id = "postgresql-primary"
42 address = "192.168.1.100"
43 port = 5432
44
45 check {
46 tcp = "192.168.1.100:5432"
47 interval = "10s"
48 }
49
50 connect {
51 sidecar_service {
52 port = 21000
53 check {
54 name = "Connect Envoy Sidecar"
55 tcp = "192.168.1.100:21000"
56 interval = "10s"
57 }
58 proxy {
59 config {
60 bind_address = "192.168.1.100"
61 bind_port = 21000
62 }
63 }
64 }
65 }
66}
1# register-vm-service.sh - Register VM-based services
2#!/bin/bash
3
4# Install Consul agent on VM
5curl -fsSL https://releases.hashicorp.com/consul/1.16.3/consul_1.16.3_linux_amd64.zip -o consul.zip
6unzip consul.zip
7sudo mv consul /usr/local/bin/
8
9# Create service definition
10cat > /etc/consul.d/vm-service.json <<EOF
11{
12 "service": {
13 "name": "legacy-app",
14 "id": "legacy-app-vm1",
15 "address": "192.168.1.101",
16 "port": 9090,
17 "tags": ["vm", "legacy"],
18 "check": {
19 "http": "http://192.168.1.101:9090/health",
20 "interval": "30s"
21 },
22 "connect": {
23 "sidecar_service": {
24 "port": 21001,
25 "check": {
26 "name": "Connect Envoy Sidecar",
27 "tcp": "192.168.1.101:21001",
28 "interval": "10s"
29 }
30 }
31 }
32 }
33}
34EOF
35
36# Start Consul agent
37consul agent -config-dir=/etc/consul.d -datacenter=dc3 \
38 -retry-join=192.168.1.100 -bind=192.168.1.101
Configure service intentions for cross-platform security:
1# Create service intentions for cross-datacenter communication
2
3# Allow web-app (K8s DC1) to communicate with api-service (Nomad DC2)
4consul intention create -allow web-app api-service
5
6# Allow api-service (Nomad DC2) to communicate with database (Bare Metal DC3)
7consul intention create -allow api-service database
8
9# Allow api-service to communicate with legacy-app (VM DC3)
10consul intention create -allow api-service legacy-app
11
12# Verify intentions
13consul intention list
Create intention configuration files for GitOps:
1# service-intentions.yaml
2apiVersion: consul.hashicorp.com/v1alpha1
3kind: ServiceIntentions
4metadata:
5 name: api-service-intentions
6 namespace: consul
7spec:
8 destination:
9 name: api-service
10 sources:
11 - name: web-app
12 action: allow
13 description: "Allow web-app to call api-service"
14 - name: "*"
15 action: deny
16 description: "Deny all other traffic"
17
18---
19apiVersion: consul.hashicorp.com/v1alpha1
20kind: ServiceIntentions
21metadata:
22 name: database-intentions
23 namespace: consul
24spec:
25 destination:
26 name: database
27 sources:
28 - name: api-service
29 action: allow
30 description: "Allow api-service to access database"
31 - name: "*"
32 action: deny
33 description: "Deny all other traffic"
Configure mesh gateways for secure cross-datacenter communication:
1# mesh-gateway-config.yaml
2apiVersion: consul.hashicorp.com/v1alpha1
3kind: ProxyDefaults
4metadata:
5 name: global
6 namespace: consul
7spec:
8 meshGateway:
9 mode: "local"
10 config:
11 protocol: "http"
12
13---
14apiVersion: consul.hashicorp.com/v1alpha1
15kind: ServiceDefaults
16metadata:
17 name: api-service
18 namespace: consul
19spec:
20 protocol: "http"
21 meshGateway:
22 mode: "local"
23
24---
25apiVersion: consul.hashicorp.com/v1alpha1
26kind: ServiceDefaults
27metadata:
28 name: database
29 namespace: consul
30spec:
31 protocol: "tcp"
32 meshGateway:
33 mode: "local"
Apply mesh gateway configurations:
1kubectl apply -f mesh-gateway-config.yaml
2kubectl apply -f service-intentions.yaml
Implement comprehensive health checking across platforms:
1# health-check-config.yaml
2apiVersion: consul.hashicorp.com/v1alpha1
3kind: ServiceDefaults
4metadata:
5 name: api-service
6 namespace: consul
7spec:
8 protocol: "http"
9 upstreamConfig:
10 defaults:
11 connectTimeoutMs: 5000
12 limits:
13 maxConnections: 50
14 maxPendingRequests: 100
15 passiveHealthCheck:
16 interval: "30s"
17 maxFailures: 3
Configure load balancing policies:
1# load-balancer-config.hcl
2Kind = "service-resolver"
3Name = "api-service"
4
5LoadBalancer = {
6 Policy = "round_robin"
7 RingHashConfig = {
8 MinimumRingSize = 1024
9 MaximumRingSize = 8192
10 }
11 HashPolicies = [
12 {
13 Field = "header"
14 FieldValue = "x-user-id"
15 }
16 ]
17}
18
19Failover = {
20 "*" = {
21 Datacenters = ["dc1", "dc3"]
22 }
23}
Implement advanced traffic management across platforms:
1# traffic-splitting.yaml
2apiVersion: consul.hashicorp.com/v1alpha1
3kind: ServiceSplitter
4metadata:
5 name: api-service
6 namespace: consul
7spec:
8 splits:
9 - weight: 80
10 service: api-service
11 serviceSubset: v1
12 - weight: 20
13 service: api-service
14 serviceSubset: v2
15
16---
17apiVersion: consul.hashicorp.com/v1alpha1
18kind: ServiceResolver
19metadata:
20 name: api-service
21 namespace: consul
22spec:
23 defaultSubset: v1
24 subsets:
25 v1:
26 filter: "Service.Tags contains v1"
27 v2:
28 filter: "Service.Tags contains v2"
29 failover:
30 "*":
31 datacenters: ["dc1", "dc3"]
Building a multi-platform service mesh with Consul Connect enables organizations to:
This architecture provides the foundation for modern hybrid and multi-cloud deployments, allowing teams to leverage the best features of each platform while maintaining security and operational consistency.
Key takeaways:
Ready to build your multi-platform service mesh? Start with establishing Consul federation between your most critical environments and gradually expand coverage to achieve complete platform unification.
For comprehensive monitoring and observability of your multi-platform service mesh, see our companion article on Advanced Monitoring and Observability for Consul Connect Service Mesh.
This is an introduction to Kubernetes, an open-source system for automating deployment, scaling, and management of containerized applications.
Introduction to HashiCorp Vault.
Connect dynamic applications.
Multi-Platform Service Mesh: Connecting Kubernetes, Nomad, Bare Metal, and VMs with Consul Connect Modern infrastructure often spans multiple platforms -
Securing Communication in Kubernetes with Istio Service Mesh and Vault Agent Injector In modern cloud-native Kubernetes environments, security is paramount. One
Securing Communication in Kubernetes with Consul Connect and Vault Agent Injector In modern cloud-native Kubernetes environments, security is paramount. One of
In a cloud-native Kubernetes environment, secrets management is a critical aspect of security. HashiCorp Vault is a popular tool for managing secrets and
Remember last July when we dug into Ollama and its promise for local AI? Well, a lot has happened since then. Ollama has rolled out some pretty significant
You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.
Contact us