CacheInvalidation.org

Understanding Redis Cache Topology

Treating Redis as a monolithic key-value repository is an architectural anti-pattern at scale. Modern deployments require engineers to view the cache as a distributed, fault-tolerant routing fabric where topology dictates horizontal scaling limits, data locality, network latency profiles, and failure domain boundaries. A properly architected Redis deployment aligns node placement, client routing logic, and eviction policies with application access patterns. While the foundational mechanics of data movement are established in Redis Caching Architecture & Invalidation Fundamentals, operational resilience depends on precise configuration tuning, deterministic routing, and automated scaling workflows that respect both physical and logical cluster boundaries.

Hash Slot Architecture and Deterministic Routing

Redis Cluster partitions the keyspace into exactly 16,384 hash slots. Each primary node owns a contiguous range of these slots, and clients must resolve slot-to-node mappings before executing commands. The mapping is deterministic: slot = CRC16(key) % 16384. This architecture eliminates single points of failure but introduces routing complexity that must be handled at the client layer.

flowchart TB
    K[key] -->|CRC16 mod 16384| SLOT[slot number]
    SLOT --> CL{Owning primary}
    CL --> N1["Node A<br/>slots 0–5460"]
    CL --> N2["Node B<br/>slots 5461–10922"]
    CL --> N3["Node C<br/>slots 10923–16383"]
    N1 --> R1[(replica)]
    N2 --> R2[(replica)]
    N3 --> R3[(replica)]

Python applications should leverage the native RedisCluster client from redis-py v4.3+. Production configurations must enable replica read routing and configure retry logic to handle cluster topology changes gracefully:

from redis.cluster import RedisCluster, ClusterNode
from redis.retry import Retry
from redis.backoff import FullJitterBackoff

startup_nodes = [
    ClusterNode("10.0.1.10", 6379),
    ClusterNode("10.0.1.11", 6379),
    ClusterNode("10.0.1.12", 6379),
]

# FullJitterBackoff(cap, base) applies full-jitter exponential backoff
retry = Retry(FullJitterBackoff(cap=2, base=0.1), retries=3)

rc = RedisCluster(
    startup_nodes=startup_nodes,
    read_from_replicas=True,
    retry=retry,
    cluster_error_retry_attempts=5,
    socket_connect_timeout=2,
    socket_timeout=2,
    decode_responses=True,
)

When a node fails or slots are migrated, the cluster returns MOVED (permanent redirection) or ASK (temporary migration in progress). The redis-py client automatically follows these redirects, but engineers must implement explicit slot cache invalidation in custom routing layers to prevent thundering herd scenarios during mass topology shifts. Setting cluster-require-full-coverage no in redis.conf allows the cluster to continue serving requests for reachable slots while marking unreachable slots as offline, shifting consistency guarantees to the application layer.

Distributed Memory Management and Eviction Calibration

In a sharded topology, memory limits are enforced per-node, not globally. A 12 GB cluster with three primaries does not guarantee 12 GB of contiguous free space; each shard independently enforces its maxmemory threshold. Misaligned eviction policies across shards cause cascading cache misses and unpredictable latency spikes.

For workloads with skewed access distributions (e.g., session stores, leaderboards, frequently accessed configuration keys), allkeys-lfu outperforms traditional LRU. LFU tracks access frequency rather than recency, preventing premature eviction of hot keys during traffic bursts. As detailed in LRU vs LFU Eviction Policies, tuning maxmemory-samples to 10 or higher improves eviction accuracy with negligible CPU overhead on modern hardware.

Production Python clients should wrap write operations with memory-aware guards to prevent OOM-induced node restarts:

from redis.cluster import RedisCluster

def safe_cluster_set(client: RedisCluster, key: str, value: str, ttl: int):
    # client.info() on a cluster returns per-node dicts; query the node
    # that owns this key via get_node_from_key().
    node = client.get_node_from_key(key)
    info = client.info("memory", target_nodes=node)
    max_mem = info.get("maxmemory") or 0  # 0 == unlimited
    usage_ratio = info["used_memory"] / max_mem if max_mem else 0.0

    if usage_ratio > 0.85:
        # Trigger targeted eviction or fall back to the primary store
        random_key = client.randomkey(target_nodes=node)
        if random_key:
            client.unlink(random_key)

    client.setex(key, ttl, value)

Monitoring used_memory_rss versus maxmemory is critical. RSS includes fragmentation overhead; if mem_fragmentation_ratio exceeds 1.2, enabling activedefrag yes or vertically scaling the node should be evaluated.

Automated Scaling and Zero-Downtime Rebalancing

Manual cluster expansion introduces human error and prolonged rebalancing windows. Automated scaling pipelines should trigger node provisioning when used_memory/maxmemory exceeds 0.75 across three consecutive polling intervals. The expansion workflow follows a deterministic sequence:

  1. Provision and Join: Deploy a new Redis instance with identical configuration and join it to the cluster as an empty primary.
  2. Assign Slots: Reshard a contiguous slot range onto the new primary.
  3. Rebalance: Migrate slots from overloaded primaries to the new node.
# 1. Add the new node as an empty primary
redis-cli --cluster add-node 10.0.1.13:6379 10.0.1.10:6379

# 2. Reshard slots onto the new primary (example: move 1365 slots ≈ 1/4 of 5461)
redis-cli --cluster reshard 10.0.1.10:6379 \
  --cluster-from <source-node-id> \
  --cluster-to <new-node-id> \
  --cluster-slots 1365 \
  --cluster-yes

Slot migration occurs incrementally. During migration, the source node returns MIGRATING state and ASK responses; the destination returns IMPORTING. The redis-py client handles ASK redirects automatically, but observability pipelines must track cluster_slots_fail and cluster_state to ensure migration completes without data loss. For large clusters, redis-cli --cluster rebalance --cluster-use-empty-masters distributes slots evenly based on current node counts.

Cross-Node Invalidation and Consistency Patterns

Broadcasting explicit DEL commands across a cluster introduces network overhead, increases latency, and creates race conditions during concurrent reads. Relying solely on TTL expiration defers consistency but risks stale data exposure. The optimal approach combines targeted invalidation with strategic TTL fallbacks, as explored in TTL vs Explicit Invalidation.

For cross-node invalidation, use Redis Pub/Sub with deterministic channel naming or pipeline-based targeted deletes:

def invalidate_across_shards(client: RedisCluster, pattern: str):
    # Publish invalidation event to a dedicated channel
    client.publish("cache:invalidation", pattern)

    # Background workers per node handle targeted deletion via scan_iter
    for key in client.scan_iter(match=pattern, count=1000):
        client.unlink(key)

To prevent race conditions during cache stampedes, implement the cache-aside with lock pattern using SET NX and short-lived TTLs. When multiple clients request the same invalidated key, only one acquires the lock to regenerate data, while others wait or serve a stale fallback.

Observability and Failure Domain Isolation

Topology awareness requires continuous telemetry. Deploy the Redis Exporter alongside Prometheus to scrape per-node metrics. Critical alerting rules include:

  • redis_cluster_slots_fail > 0 — Immediate paging
  • redis_memory_used_bytes / redis_memory_max_bytes > 0.80 — Scale trigger
  • redis_keyspace_misses_total / (redis_keyspace_hits_total + redis_keyspace_misses_total) > 0.3 — Eviction misconfiguration
  • redis_cluster_state != "ok" — Topology degradation

Multi-tenant deployments require strict failure domain isolation. Logical separation via key prefixes is insufficient for noisy-neighbor scenarios. Instead, deploy dedicated Redis clusters per tenant tier or leverage Redis ACLs with resource quotas. Security boundaries must enforce network segmentation, TLS encryption, and command allowlisting to prevent cross-tenant data leakage, as outlined in Redis Security Boundaries for Multi-Tenant Apps.

For comprehensive cluster management, consult the official Redis Cluster Specification and integrate redis-cli --cluster check into CI/CD pipelines to validate topology health before deployments.

Operational Checklist

Topology-aware caching transforms Redis from a volatile storage layer into a predictable, horizontally scalable routing fabric. By aligning client configuration, memory policies, and automation pipelines with the underlying hash slot architecture, engineering teams achieve consistent latency, graceful degradation during failures, and seamless capacity expansion.