Rare-Event Memory Updating for Low-Frequency Threat Discovery in Streaming Security Logs

Abstract

Security operation centers receive continuous log streams from firewalls, endpoint agents, authentication systems, cloud services, and intrusion prevention systems. Low-frequency anomalies are difficult to detect because they may appear as weak signals hidden among repeated benign events and gradually changing operational patterns. This study proposes a cluster memory regulation method for detecting low-frequency anomalies in streaming security logs. The method uses online clustering to summarize normal log-event behavior and introduces a memory-retention coefficient to preserve sensitivity to rare deviations. A log-template embedding module converts heterogeneous event messages into structured vectors, while a time-aware deviation score measures changes in event frequency, source identity, destination service, and action sequence. Experiments are conducted on an enterprise security-log stream containing 8.4 billion log entries collected from 18,200 endpoints, 1,140 servers, 76 firewall zones, and 320 cloud service accounts over 60 days. After template parsing, 4.9 million unique event sequences are mapped into streaming feature vectors. The dataset includes 3,760 annotated anomaly episodes, including abnormal login bursts, rare privilege-use sequences, suspicious outbound access, policy-denied activity surges, and endpoint process anomalies. The proposed method reduces median detection delay from 21.4 minutes to 6.8 minutes compared with a sliding-window isolation forest. False alarms are controlled at 5.2 investigation cases per 10 million log events. The system processes 156,000 parsed log events per second and maintains 12,600 active cluster states with 1.9 GB memory usage. Memory regulation prevents 1,430 rare anomaly patterns from being prematurely merged into normal clusters during long-running monitoring. The results demonstrate that cluster memory regulation can enhance streaming log anomaly detection under sparse and evolving threat conditions.