Skip to content

Redis.md

Latest commit

 

History

History
32 lines (19 loc) · 3.2 KB

Redis.md

File metadata and controls

32 lines (19 loc) · 3.2 KB
Raw

Redis Usage in this System: Architecture and Best Practices

Redis Core Usage

  • Caching of Datasets: Redis serves as an in-memory cache for datasets (e.g., MNIST) used during model training, avoiding repeated access to Cassandra and improving performance.
  • Data Retrieval Optimization: The system first checks Redis when fetching datasets. If the data is missing, it fetches the data from Cassandra and stores it in Redis for future requests.

Design Considerations

  • Performance Optimization: Redis reduces I/O overhead by storing frequently accessed data in memory, minimizing the need to query Cassandra repeatedly.

  • Low Latency Access: As an in-memory store, Redis offers sub-millisecond access times, supporting real-time tasks like log streaming and feedback delivery.

  • Scalability and Fault Tolerance: Although Redis is used primarily as a cache, it reduces the load on Cassandra by serving as a buffer. This design ensures the system scales effectively.

  • Decoupling Components: By using Redis as a caching layer, the system remains decoupled—the training process can proceed even if Cassandra is temporarily unavailable.

Best Practices in this System

  • Efficient Caching Strategy with Pickle Serialization : Use pickle serialization to store complex Python objects (e.g., NumPy arrays) in Redis. Ensures compatibility between Redis and Python, making it easy to store and retrieve complex datasets.
  • Cache Validation and Fallback Mechanism: Check Redis first for datasets, and if missing, fetch from Cassandra and cache the result in Redis. Reduces the number of database queries, optimizing resource usage and maintaining data availability.
  • Session-level Data Management for Memory Efficiency : Use expiration policies on Redis keys or manually remove them to prevent memory bloat. Ensures that cached data only persists for active sessions, keeping memory usage efficient.
  • Real-time Logging with Kafka Integration: Log Redis operations (cache hits/misses) and send them to Kafka topics, allowing real-time updates on the frontend using Server-Sent Events (SSE).Provides transparency and visibility into the system’s behavior, enhancing the user experience with up-to-date information.
  • Graceful Handling of Cache Misses: If Redis cache misses occur, fetch data from Cassandra and handle failures by logging errors or raising HTTP exceptions. Ensures smooth error handling and system reliability, even in cases where the cache is not populated.
  • Redis Client Configuration for Scalability: Initialize the Redis client once at the application level to reuse connections across multiple requests. Reduces connection overhead and ensures the system can handle high concurrency efficiently.

Conclusion

Redis plays a critical role in this system by acting as a high-performance cache, reducing the load on Cassandra, and enabling fast retrieval of datasets. By using efficient caching strategies with pickle serialization and integrating with Kafka for real-time logging, the system achieves both high scalability and transparency. Redis ensures that model training sessions proceed smoothly with low-latency data access, supporting the overall architecture’s performance goals.