P15PythonAdvancedImplement16 min
Process a Large Stream Without Loading Everything Into Memory
Design a memory-safe approach for a multi-gigabyte clickstream file.
#generators#streaming#memory#aggregation
Scenario
A daily clickstream file is too large to load into memory, but operations needs counts by event type and hour.
Problem statement
Describe a streaming design that produces reliable aggregates and can recover from malformed lines.
Your Task
- Process one record or chunk at a time.
- Keep only aggregate state in memory.
- Decide how malformed records are logged and monitored.
Related Concepts
IteratorsFault toleranceStreaming aggregation
Ready to count this problem?
Mark it complete when you have reasoned through the solution in your own words.