// DISTRIBUTED LOCKING

Quorum-Based Consensus

Coordinate concurrent operations safely across your cluster. Prevent race conditions and ensure only one critical operation runs at a time.

Real-World Use Case: Database Migration

You're migrating a database schema across 10 application servers. Only ONE server should run the migration at a time. Quorum-based locking ensures this.

1. Quorum Consensus

Locking API

# Acquire lock (requires majority agreement)
$ curl -X POST /api/v1.0/lock \
  -d '{
  "scope": "db-migration",
  "action": "lock"
}'

{
  "status": "locked",
  "scope": "db-migration",
  "quorum": "6/10 nodes agreed",
  "holder": "migration-orchestration-001"
}

# Release lock
$ curl -X POST /api/v1.0/lock \
  -d '{
  "scope": "db-migration",
  "action": "unlock"
}'

How It Works

Majority of nodes must agree (quorum)
Prevents split-brain scenarios
Automatic release on timeout
No single point of failure

Unlike Kubernetes' single etcd master, this is truly distributed.

2. DShell CLI - Interactive Lock Management

Operational Control via DShell

Interactive shell for managing locks without writing code:

manager lock acquire --scope SCOPE
manager lock status --scope SCOPE
manager lock release --scope SCOPE
Useful for operational teams and debugging

DShell provides a clean interface for HTTP APIs. Perfect for ops teams who prefer CLI over REST.

DShell Locking Commands

# Start interactive dshell
$ dshell -s cluster-node-1 -p 20194 -u admin

# Acquire distributed lock
dshell> manager lock acquire --scope db-migration
Lock acquired. Holder: dshell-session-001

# Check lock status
dshell> manager lock status --scope db-migration
LOCKED by dshell-session-001 (5/7 nodes)

# Release lock
dshell> manager lock release --scope db-migration
Lock released

# Batch mode for scripts
$ dshell manager lock acquire --scope db-migration

3. Orchestration Integration

Orchestration with Locking

{
  "name": "safe-db-migration",
  "steps": [
    {
      "name": "acquire-lock",
      "action_type": "REQUEST",
      "code": "{\"method\": \"POST\", \"url\": \"https://node/api/v1.0/lock\", \"json\": {\"scope\": \"db-migration\", \"action\": \"lock\"}}",
      "expected_rc": 200,
      "id": "1",
      "undo": false,
      "target": "all",
      "parent_step_ids": []
    },
    {
      "name": "run-migration",
      "action_type": "SHELL",
      "code": "alembic upgrade head",
      "expected_rc": 0,
      "id": "2",
      "undo": false,
      "target": "all",
      "parent_step_ids": [
        "1"
      ]
    },
    {
      "name": "release-lock",
      "action_type": "REQUEST",
      "code": "{\"method\": \"POST\", \"url\": \"https://node/api/v1.0/lock\", \"json\": {\"scope\": \"db-migration\", \"action\": \"unlock\"}}",
      "expected_rc": 200,
      "id": "3",
      "undo": false,
      "target": "all",
      "parent_step_ids": [
        "2"
      ]
    }
  ]
}

Lock within Orchestrations

Use locks to coordinate multi-step operations:

Acquire lock at step start
Run critical operations
Release lock on completion
Auto-release on failure

Additional Capabilities

🔄 Read-Write Lock Patterns

Support both exclusive locks and shared read locks. Allow multiple readers, but exclusive access for writers.

Lock Types

"lock_type": "read",
"scope": "config-db",
"holder": "reader-1"
// Multiple readers
"lock_type": "write"
// Only one writer

⏱️ Adaptive Lock Timeouts

Automatically adjust lock timeout based on operation complexity and historical execution time.

Timeout Config

"timeout": "auto",
"min_timeout": 5s,
"max_timeout": 300s,
"learning_window": "7d"

📊 Lock Contention Metrics

Monitor lock contention, waiter queue, and hold duration for performance optimization.

Metrics

"contention": {
  "waiters": 3,
  "hold_duration": "2.3s",
  "conflict_rate": "12%"
}

🎯 Priority-Based Lock Acquisition

Assign priorities to lock requests. High-priority operations skip the queue in emergency situations.

Priority Levels

"priority": "high",
"reason": "incident_recovery",
"queue_position": "1"
"normal" / "low" also available

🔐

Safe Concurrency

Prevent race conditions without a central coordinator.

🤝

Consensus-Based

Quorum agreement ensures correctness even if nodes fail.

⏱️

Timeout Protection

Locks auto-release if holder crashes.

📍

Scope-Based

Multiple independent locks for different resources.

Explore Mesh Networking Back to Features