Unit II: Key-Value Databases - Redis#

What is a Key-Value Database?#

A key-value database stores data as pairs of a unique key and its associated value — exactly like a dict in Python or a HashMap in Java.

Analogy — Gym Locker System:

Think of a gym locker room. Each locker has a unique number (the KEY) and whatever you store inside is the VALUE. If you know the locker number, you retrieve your belongings in O(1) time — no searching through every locker.

KEY                          VALUE
─────────────────────────    ──────────────────────────────
user:1001:name           →   "Alice"
session:abc123           →   "{userId: 1001, role: admin}"
product:iphone15:price   →   "1299"
leaderboard              →   [sorted list of players]

Redis Architecture#

Redis has 3 main layers: Client, Redis Server, and Storage & Replicas.

1. Clients (Who talks to Redis)

Three types of apps send requests to Redis:

• Web App
• Mobile App
• Microservice All three connect using TCP/IP on port 6379 — the default Redis port.

2. Redis Server (The brain) Every incoming request first hits the Command Parser. Think of it as a receptionist — it reads the command (like GET key or SET key value) and figures out what to do with it.

Then the parsed command goes straight to the In-Memory Hash Table — this is where all the actual data lives, stored in RAM for lightning-fast access.

3. Three Engines (What happens after data is stored) From the Hash Table, data flows to three systems: EngineJobReplication ManagerCopies data to replica servers for backup/load balancingPersistence EngineSaves data to disk so it survives restartsPub/Sub EngineHandles real-time messaging between publishers and subscribers

4. Storage & Replicas (The bottom) Replicas (left side):

Replica 1 & Replica 2: exact copies of the main Redis data, kept in sync by the Replication Manager

Storage (right side): two ways Redis saves data to disk:

RDB Snapshot (.rdb file): periodic full snapshots, like a photo of the data at a point in time AOF Log (.aof file) — logs every write command, like a receipt history. More detailed but larger

The Key Idea

• Redis keeps everything in memory (RAM) for speed, but uses persistence + replication to make sure data isn’t lost if something crashes.

How Redis Stores Data Internally#

Redis uses a hash table internally — the same data structure as dict in Python. This gives O(1) average time for GET and SET.

Real-World Example — Twitter/X: Twitter uses Redis to store each user’s timeline. When we open the app, it doesn’t run a complex SQL JOIN across 500 million tweets — it simply fetches a pre-built list stored in Redis under timeline:user:1234. Sub-millisecond response time for 300M+ users.

# How Redis thinks of your data internally:
{
    "user:1001:name"    : "Alice",
    "user:1001:email"   : "alice@example.com",
    "product:42:stock"  : "150",
    "session:xyz789"    : '{"userId": 1001, "role": "admin"}',
    "counter:visits"    : "94827"
}

Advantages#

Limitations#

Key Insight for Students: Redis is NOT a replacement for PostgreSQL or MySQL. It’s a complement. Use Redis as a cache, session store, queue, or leaderboard — alongside a persistent relational database.

Key Naming Conventions#

Since all keys share the same namespace, structured naming is critical:

Pattern: object-type:id:attribute

Examples:
user:1001:profile          # User profile data
user:1001:sessions         # User's active sessions
product:sku:BTN500:details # Product details
order:ORD2024001:status    # Order status
cache:homepage:trending    # Cached content
ratelimit:user:1001:api    # Rate limiting counter

Tip: Use colons (:) as separators consistently. This lets you use SCAN 0 MATCH user:* to find all user-related keys safely in production.

Key Rules#

1. Strings#

The most fundamental type. Binary-safe. Can hold text, numbers, serialized JSON, even images. Max: 512 MB.

SET username "Alice"
SET counter 0
SET user:1001:profile '{"name":"Alice","age":25}'   # Store JSON as string
SET flag 1

GET username                    # Returns "Alice"
GET counter                     # Returns "0"
INCR counter                    # counter → 1 (ATOMIC increment)
INCRBY counter 5                # counter → 6
DECR counter                    # counter → 5
INCRBYFLOAT price 1.5           # Float increment

# Key with expiry (TTL):
SETEX session:tok123 3600 "user:1001"   # Expires in 3600 seconds
TTL session:tok123              # Returns remaining seconds
PTTL session:tok123             # Returns remaining milliseconds
EXPIRE username 86400           # Set expiry on existing key
PERSIST username                # Remove expiry (make permanent)

Real-World Example — Rate Limiting (Twitter): Twitter limits 300 tweets per 3 hours. Implementation:

SET ratelimit:user:1001:tweets 0
INCR ratelimit:user:1001:tweets   # On each tweet
EXPIRE ratelimit:user:1001:tweets 10800   # 3 hours
# When value hits 300 → reject the request

2. Lists#

An ordered list of strings, implemented as a doubly linked list. Supports push/pop from both ends — perfect for queues and stacks.

Left End (Head)                          Right End (Tail)
      ▼                                        ▼
LPUSH/LPOP  [resize_image] [send_email] [gen_pdf]  RPUSH/RPOP

← QUEUE  (LPUSH + RPOP) →
←← STACK (LPUSH + LPOP) →

LPUSH tasks "send_email"          # Push to LEFT: ["send_email"]
LPUSH tasks "resize_image"        # ["resize_image", "send_email"]
RPUSH tasks "generate_pdf"        # ["resize_image", "send_email", "generate_pdf"]
LLEN tasks                        # Returns 3
LRANGE tasks 0 -1                 # Get ALL elements (0=first, -1=last)
LRANGE tasks 0 1                  # Get first 2 elements
LINDEX tasks 0                    # Get element at index 0
LPOP tasks                        # Remove & return "resize_image" (from left)
RPOP tasks                        # Remove & return "generate_pdf" (from right)

# Blocking pop — WAITS for a new task (perfect for workers!)
BLPOP tasks 30                    # Block up to 30 seconds waiting for a task

Real-World Example — Celery Task Queue (Instagram): When a user uploads a photo on Instagram, a job is pushed to a Redis list: RPUSH jobs:image:resize "{photoId: 9876}". Background worker processes (using BLPOP) pick up and process jobs — resizing images, sending notifications — without polling.

3. Sets#

An unordered collection of unique strings. Duplicates are automatically ignored. Supports powerful math operations.

SADD followers:user:1001 "user:2001" "user:3001" "user:4001"
SADD following:user:2001 "user:1001" "user:5001"

SMEMBERS followers:user:1001              # Get all members
SISMEMBER followers:user:1001 "user:2001" # Returns 1 (is member)
SCARD followers:user:1001                 # Count = 3
SPOP followers:user:1001                  # Remove & return a random member

# ─── Set Math Operations ───
SADD tags:article:101 "redis" "nosql" "database"
SADD tags:article:102 "redis" "caching" "performance"

SINTER tags:article:101 tags:article:102  # Intersection → {"redis"}
SUNION tags:article:101 tags:article:102  # Union → all unique tags
SDIFF  tags:article:101 tags:article:102  # Difference → {"nosql","database"}

Real-World Example — LinkedIn “People You May Know”:

SINTER connections:alice connections:bob
# Returns users known by BOTH → potential connection suggestions

4. Hashes#

A field-value map stored under a single key. Think: a mini-dictionary, or a single database row. Perfect for representing objects.

Hash vs String — Which to use for objects?

# Store a user object as a Hash
HSET user:1001 name "Alice Sharma" email "alice@example.com" age 28 city "Thimphu"

HGET user:1001 name             # Returns "Alice Sharma"
HGET user:1001 email            # Returns "alice@example.com"
HGETALL user:1001               # Returns ALL fields and values
HMGET user:1001 name email      # Get multiple specific fields
HKEYS user:1001                 # Returns field names: [name, email, age, city]
HVALS user:1001                 # Returns values only
HLEN user:1001                  # Returns 4 (number of fields)
HSET user:1001 name "Alice S."  # Update just one field
HDEL user:1001 city             # Delete one field
HEXISTS user:1001 email         # Does field exist? → 1
HINCRBY user:1001 age 1         # Atomically increment age field

Real-World Example — GitHub Repositories:

HSET repo:torvalds/linux stars 170000 forks 49000 language "C"

When you open the repo page, GitHub fetches all fields with HGETALL in microseconds — no SQL query.

5. Sorted Sets (ZSets)#

The most powerful Redis type. Like a Set, but every member has a floating-point score. Members are always kept sorted by score. Time complexity: O(log N) for most operations.

# ─── Game Leaderboard ───
ZADD leaderboard 9500 "player:alice"
ZADD leaderboard 8200 "player:bob"
ZADD leaderboard 9800 "player:charlie"
ZADD leaderboard 7100 "player:diana"

# Get top 3 (highest scores first):
ZREVRANGE leaderboard 0 2 WITHSCORES
# → charlie 9800 | alice 9500 | bob 8200

# Get rank (0-indexed, best = rank 0):
ZREVRANK leaderboard "player:alice"         # → 1 (2nd place)
ZSCORE leaderboard "player:alice"           # → 9500.0

# Add points:
ZINCRBY leaderboard 500 "player:alice"      # alice → 10000

# Range by score:
ZRANGEBYSCORE leaderboard 8000 10000 WITHSCORES  # All between 8000-10000
ZCOUNT leaderboard 8000 +inf                # Count with score >= 8000

# Remove lowest scorer:
ZPOPMIN leaderboard                         # Removes & returns diana (lowest score)

LEADERBOARD (sorted by score, desc):
┌────────────────────────────┐
│ 🥇 charlie      │  9800   │
│ 🥈 alice        │  9500   │
│ 🥉 bob          │  8200   │
│    diana        │  7100   │
└────────────────────────────┘
ZREVRANGE leaderboard 0 -1 WITHSCORES

Real-World Example — Stack Overflow: ZREVRANGE reputations 0 99 WITHSCORES — fetches the top 100 users for the “Top Users” page in real-time. No complex SQL, no sorting at query time.

Global Key Commands (work on ALL types)#

EXISTS user:1001          # Does key exist? → 1 or 0
DEL user:1001             # Delete key
TYPE user:1001            # Returns: string | list | set | hash | zset
RENAME user:1001 user:old:1001
EXPIRE user:1001 3600     # Set TTL (seconds from now)
EXPIREAT user:1001 1735689600  # TTL = Unix timestamp
PEXPIRE user:1001 60000   # TTL in milliseconds
TTL user:1001             # Remaining TTL in seconds (-1 = no TTL, -2 = key gone)
PERSIST user:1001         # Remove TTL

# ─── Searching Keys ───
KEYS user:*               # DANGEROUS in production!
SCAN 0 MATCH user:* COUNT 100  # Safe cursor-based iteration

WARNING — Never KEYS * in Production! KEYS is O(N) and blocks ALL other operations while scanning. Use SCAN instead — it iterates in small batches without blocking. This is a common mistake that can bring down production Redis.

Server Commands#

PING               # Test connection → PONG
INFO               # Full server stats
INFO memory        # Memory-specific stats
INFO replication   # Replication status
DBSIZE             # Count total keys
CONFIG GET maxmemory       # Read config value
CONFIG SET maxmemory 2gb   # Update config live (no restart!)
SLOWLOG GET 10             # Last 10 slow commands
FLUSHDB                    # Delete ALL keys in current DB
FLUSHALL                   # Delete ALL keys in ALL DBs

Bitmaps#

# ─── Daily User Login Tracking ───
# Bit position = User ID
SETBIT logins:2024-01-15 1001 1     # User 1001 logged in today
SETBIT logins:2024-01-15 2005 1     # User 2005 also logged in
SETBIT logins:2024-01-15 3099 1

GETBIT logins:2024-01-15 1001       # → 1 (logged in)
GETBIT logins:2024-01-15 9999       # → 0 (did NOT log in)
BITCOUNT logins:2024-01-15          # Total logins today

# Who was active on BOTH Jan 15 and Jan 16?
BITOP AND active_both logins:2024-01-15 logins:2024-01-16
BITCOUNT active_both                # Users active on both days

Why Bitmaps are a Game-Changer:

Bitfields#

Store integers of custom bit-widths in a compact binary format:

# Compact game player stats:
# Offset 0:  level (u8,  0-255)
# Offset 8:  health (u8, 0-255)
# Offset 16: gold (u16, 0-65535)

BITFIELD player:1001 SET u8  0  15    # Level  = 15
BITFIELD player:1001 SET u8  8  87    # Health = 87
BITFIELD player:1001 SET u16 16 5000  # Gold   = 5000

BITFIELD player:1001 GET u8 0         # → 15
BITFIELD player:1001 INCRBY u8 0 1    # Level up! → 16

# Overflow protection: SAT = saturate at max (no wrap-around)
BITFIELD player:1001 OVERFLOW SAT INCRBY u8 8 200  # Health stays at 255

Real-World Example — GitHub Contribution Graph: Those green squares on GitHub profiles? Each bit represents one day. BITCOUNT contributions:user:torvalds = total active days. Storing 365 days of activity for 100M users = ~4.5 GB with bitmaps (vs ~365 GB with a Set per user).

Hash Slot Sharding#

              16,384 Hash Slots
                     │
    ┌────────────────┼────────────────┐
    ▼                ▼                ▼
🖥 Node A        🖥 Node B        🖥 Node C
Slots 0–5460   Slots 5461–10922  Slots 10923–16383

key: 'user:1001'    → CRC16 % 16384 = 7638  → Node B
key: 'session:abc'  → CRC16 % 16384 = 2891  → Node A
key: 'product:55'   → CRC16 % 16384 = 12001 → Node C

# Hash slot formula:
slot = CRC16(key) % 16384

# ─── Force keys to same slot (Hash Tags) ───
{user:1001}.name     # Both hashed on "user:1001" → same slot
{user:1001}.email    # → same node → multi-key ops work!

# redis.conf for each cluster node:
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000

# Create cluster (3 masters + 3 replicas):
redis-cli --cluster create \
  192.168.1.101:6379 192.168.1.102:6379 192.168.1.103:6379 \
  192.168.1.104:6379 192.168.1.105:6379 192.168.1.106:6379 \
  --cluster-replicas 1

redis-cli -c cluster info    # Status
redis-cli -c cluster nodes   # Node details

Sentinel vs Cluster Comparison#

Real-World Example — Instagram: Instagram uses Redis Cluster across thousands of nodes to store follower/following relationships and feed data for 1+ billion users. No single machine could hold all this in RAM.

A. The Index Creation: FT.CREATE#

FT.CREATE idx:products
  ON HASH
  PREFIX 1 product:
  SCHEMA
    name        TEXT WEIGHT 5.0   # Higher weight = more relevant in search
    description TEXT
    price       NUMERIC SORTABLE
    category    TAG
    brand       TAG SORTABLE

Command Breakdown:

• idx:products — Identifier for the search index
• ON HASH — Tracks Redis Hashes (field-value pairs)
• PREFIX 1 product: — Only indexes keys starting with product:
• SCHEMA — Declares which fields are searchable

Field Types:

Special Modifiers:

• WEIGHT <value> — Assigns importance (e.g., 5.0 on name ranks name matches higher)
• SORTABLE — Keeps metadata in memory for SORTBY clauses

B. Data Ingestion: HSET#

HSET product:1001 name "iPhone 15 Pro" description "Apple flagship" price 1299 category "smartphone" brand "Apple"
HSET product:1002 name "Samsung Galaxy S24" description "Android flagship" price 999 category "smartphone" brand "Samsung"
HSET product:1003 name "Sony WH-1000XM5" description "Noise cancelling" price 350 category "headphones" brand "Sony"

The engine automatically detects when a new key matching product: is created and adds it to the index in the background.

C. Querying: FT.SEARCH#

FT.SEARCH idx:products "iPhone"                          # Full-text search
FT.SEARCH idx:products "@category:{smartphone}"          # Tag filter
FT.SEARCH idx:products "@price:[200 500]"                # Numeric range
FT.SEARCH idx:products "flagship @price:[1000 +inf]"     # Combined
FT.SEARCH idx:products "*" SORTBY price ASC              # Sort by price
FT.SEARCH idx:products "iPhone" LIMIT 0 10               # Pagination

# ─── Aggregations ───
FT.AGGREGATE idx:products "*"
  GROUPBY 1 @category
  REDUCE COUNT 0 AS product_count

Query Pattern Reference:

• @category:{smartphone} — @field + {} for TAG fields
• @price:[200 500] — Inclusive numeric range
• +inf — Represents infinity in numeric ranges
• LIMIT 0 10 — Like SQL OFFSET 0 LIMIT 10

D. Key Takeaways#

1. Decoupling — Index is separate from data. Deleting the index leaves Hash data intact.
2. Performance — Uses Inverted Index → O(1) or O(log N) vs O(N) for full scans.
3. Real-time — Index updates immediately on HSET or DEL on matching prefix.

Real-World Example — E-Commerce Search: When you search “iPhone 15 case black” on an e-commerce site, RediSearch returns ranked results in under 10ms, combining full-text matching with faceted filtering (brand, price range, category).

Pipelining#

WITHOUT Pipelining (100 round-trips):
  SET key:1 → OK
  SET key:2 → OK
  ... (100 times)

WITH Pipelining (1 round-trip):
  SET key:1 | SET key:2 | ... (100 commands) → OK OK OK ...

# Python example (redis-py library):

# Without pipelining — 10,000 round-trips
import redis
r = redis.Redis()
for i in range(10000):
    r.set(f"key:{i}", f"value:{i}")   # Slow!

# With pipelining — ~1 round-trip
pipe = r.pipeline()
for i in range(10000):
    pipe.set(f"key:{i}", f"value:{i}")   # Just queues commands
results = pipe.execute()               # Send all at once!
# ~100x faster! 🚀

Transactions (MULTI/EXEC)#

Redis ≠ SQL Transactions. Redis does NOT roll back on runtime errors. If command 3 of 5 fails, commands 1, 2, 4, 5 still execute. MULTI/EXEC only guarantees no interleaving, not rollback.

# ─── Transfer credits atomically ───
MULTI                             # Start transaction (queue mode)
DECRBY user:1001:credits 100      # Queued
INCRBY user:2001:credits 100      # Queued
EXEC                              # Execute ALL atomically
DISCARD                           # Cancel transaction

# ─── Optimistic Locking with WATCH ───
WATCH user:1001:credits           # Monitor for changes
credits = GET user:1001:credits
MULTI
DECRBY user:1001:credits 100
EXEC    # Returns nil if credits changed since WATCH → retry!

Lua Scripts (Atomic Complex Operations)#

# Atomic check-then-deduct:
EVAL "
  local balance = redis.call('GET', KEYS[1])
  if tonumber(balance) >= tonumber(ARGV[1]) then
    redis.call('DECRBY', KEYS[1], ARGV[1])
    return 1   -- success
  else
    return 0   -- insufficient funds
  end
" 1 user:1001:credits 100

# Pre-load script (use SHA hash to avoid re-sending):
SCRIPT LOAD "return redis.call('GET', KEYS[1])"   # → returns SHA1
EVALSHA <sha1_hash> 1 somekey

Eviction Policies#

Memory Optimization Tips#

# Check memory usage of a key:
MEMORY USAGE user:1001          # Returns bytes

# Check internal encoding:
OBJECT ENCODING user:1001       # ziplist / listpack / hashtable / etc.

# Memory analysis:
INFO memory
# Key fields to watch:
# used_memory_human      → current allocated
# mem_fragmentation_ratio → >1.5 = high fragmentation (consider restart)
# maxmemory_human        → configured limit

MEMORY DOCTOR               # Auto-diagnosis and suggestions

💡 Pro Tip — Hash for Small Objects: Redis uses a compact listpack encoding for hashes with fewer than 128 fields and values under 64 bytes. Storing 1M user objects as hashes can be 10x more memory-efficient than using separate string keys.

Key Metrics to Monitor#

Legacy Password (Redis < 6.0)#

# redis.conf:
requirepass YourStr0ngP@ssw0rd!

# Connect:
redis-cli -h localhost -p 6379 -a YourStr0ngP@ssw0rd!

# Or after connecting:
AUTH YourStr0ngP@ssw0rd!

ACL — Access Control Lists (Redis 6.0+) Modern Approach#

# ─── Create users with specific permissions ───
ACL SETUSER alice on >alice_pass ~user:* +GET +HGET +HGETALL
# alice: enabled | password | key pattern | allowed commands

ACL SETUSER bob on >bob_pass ~order:* +@read
# bob: can read any key matching order:*

ACL SETUSER api_service on >svc_pass ~cache:* +GET +MGET +LRANGE +SETEX
# Microservice: read-only access to cache keys

ACL SETUSER admin on >admin_pass ~* +@all
# Full access admin

# ─── Disable default (unauthenticated) user ───
ACL SETUSER default off

# ─── Inspect ACLs ───
ACL LIST              # Show all rules
ACL WHOAMI            # Who am I currently?
ACL CAT               # All command categories
ACL CAT string        # All string commands
ACL LOG               # Security events (failed auths, denied commands)

ACL Command Categories#

The Redis Security Checklist#

🚨 Real Security Incident — The Crackit Attack (2016): Tens of thousands of Redis instances were exposed to the internet with no auth and no firewall. Attackers used CONFIG SET dir /root/.ssh + CONFIG SET dbfilename authorized_keys + SET to write their SSH key, gaining full root server access. Always firewall your Redis port. Always.