Redis Engineering Guide

Architecture, Implementation, Management, and Python Integration for Quantitative Pipelines

1. The Redis Advantage: Architecture & Performance

Redis (REmote DIctionary Server) is an open-source, in-memory data structure store designed for lightning-fast caching, messaging, and database workloads.

• Written in C: Stripped of heavy runtime overhead, its core engine operates with maximal efficiency, delivering sub-millisecond response times and handling hundreds of thousands of operations per second.
• Data Structures (Hashes & JSON): Beyond simple strings, Redis supports robust data structures like Hashes. A Redis Hash operates almost identically to a Python dictionary, allowing you to map multi-field records (such as option snapshots and stock summaries) directly without complex serialization.
• Low Memory Footprint: Keeping data entirely in RAM using optimized memory allocators ensures massive datasets compress down to tiny footprints—typically consuming only a few megabytes for thousands of structured records.
• Robustness & High Availability: Built to be a permanent infrastructure fixture with persistence options (RDB snapshots and AOF logs) and replication architectures like Sentinel to guarantee uptime.

2. Installation on Linux and Raspberry Pi

Installing Redis on Debian/Ubuntu Linux distributions (including Raspberry Pi OS) is fast and reliable via the native package manager:

sudo apt update
sudo apt install redis-server -y
sudo systemctl enable redis-server
sudo systemctl start redis-server

To verify service health:

sudo systemctl status redis-server

3. Interactive Command Line Interface (redis-cli) for Quick Checks & Maintenance

The redis-cli utility is indispensable for inspecting data, running ad-hoc queries, and handling maintenance:

• Connect locally: redis-cli
• Connect remotely with a password: redis-cli -h 192.168.1.64 -a "changemeNow"
• Test connectivity (Ping): PING → PONG
• Inspect a Hash record: HGETALL option:AVGO:20220730:450:70:1494
• Search keys using patterns: KEYS option:AVGO*
• Check memory footprint: MEMORY STATS or info memory | grep used_memory_human
• Check database size: DBSIZE

4. Connection Management & decode_responses=True

By default, Redis stores and returns all keys and values in raw binary format (bytes) to preserve memory. In Python, this requires manual decoding unless configured otherwise.

The Golden Rule: Always instantiate your Python client with decode_responses=True:

import redis

client = redis.Redis(
    host="192.168.1.64",
    port=6379,
    password="changemeNow",
    decode_responses=True
)

5. High Availability: Passwords and Replication (Redis Sentinel)

For production environments requiring high availability, pair a master Redis instance with replicas monitored via Redis Sentinel, which automatically handles failover and master promotion.

import redis
from redis.sentinel import Sentinel

SENTINEL_NODES = [('192.168.1.64', 26379), ('192.168.1.82', 26379)]
REDIS_PASSWORD = "changemeNow"

sentinel = Sentinel(SENTINEL_NODES, socket_timeout=0.2, sentinel_kwargs={"password": REDIS_PASSWORD})
master_client = sentinel.master_for('mymaster', socket_timeout=0.2, password=REDIS_PASSWORD, decode_responses=True)

6. Migrating Records Between Instances / Databases

To copy data across instances or database indexes programmatically:

import redis

src_client = redis.Redis(host="192.168.1.64", port=6379, password="changemeNow", decode_responses=True)
dst_client = redis.Redis(host="localhost", port=6379, decode_responses=True)

for key in src_client.scan_iter("option:*"):
    key_type = src_client.type(key)
    if key_type == "hash":
        dst_client.hset(key, mapping=src_client.hgetall(key))
    elif key_type == "string":
        dst_client.set(key, src_client.get(key))
    
    ttl = src_client.ttl(key)
    if ttl > 0:
        dst_client.expire(key, ttl)

7. Managing Dev vs. Prod in config.py (Lazy Loading & Explicit Clients)

To prevent import-order clashes where helper modules eagerly connect before environment overrides take effect, use lazy loading and explicit client passing:

import os
import socket
import redis
from redis.sentinel import Sentinel

def get_redis_client(target_env=None):
    ENV = target_env or os.getenv("ENV")
    if not ENV:
        ENV = "dev" if socket.gethostname() == "coeus" else "prod"
    ENV = ENV.lower()

    if ENV in ("prod", "production"):
        PRIMARY_IP = "192.168.1.64"
        REDIS_PASSWORD = "changemeNow"
        SENTINEL_NODES = [('192.168.1.64', 26379), ('192.168.1.82', 26379)]
        try:
            client = redis.Redis(host=PRIMARY_IP, port=6379, password=REDIS_PASSWORD, decode_responses=True, socket_timeout=0.2)
            client.ping()
            return client
        except (redis.ConnectionError, redis.TimeoutError):
            sentinel = Sentinel(SENTINEL_NODES, socket_timeout=0.2, sentinel_kwargs={"password": REDIS_PASSWORD})
            return sentinel.master_for('mymaster', socket_timeout=0.2, password=REDIS_PASSWORD, decode_responses=True)
    elif ENV in ("dev", "devel", "sandbox", "sbox"):
        return redis.Redis(host="localhost", port=6379, password=None, decode_responses=True)
    else:
        raise ValueError(f"Unsupported ENV value: '{ENV}'")

def chk_redis(client=None, prnt=False):
    active_client = client if client is not None else get_redis_client()
    host = active_client.connection_pool.connection_kwargs.get("host")
    port = active_client.connection_pool.connection_kwargs.get("port")
    print(f"Connected to Redis host: {host}:{port}")
    return (host, port)

Written with assistance from Google: gemini