The Explainable Deep Learning Approach: Bridging Detection and Root Cause Analysis for Distributed Log Anomalies using TLAN-EX

Abstract

Looking deeper into modern distributed banking environments, it is discovered that nowadays, distributed Log is facing challenges because transaction data comes from multiple networked nodes, which makes a complete system behavior check difficult. This shows how complications grow when multiple system components produce logs at the same time, because this gives rise to data inconsistencies that cause anomaly detection to be more difficult. The detection systems we use nowadays show lower reliability because of high log volumes and logs that don’t follow the standard format, and events showing no clear order, due to which the system raises alarms and fails to catch the real anomalies. Research into log anomaly detection by using deep learning models has shown positive results, but distributed systems make it hard and difficult for real-life implementation. The current sequence models used in anomaly detection systems only identify one pattern type at a time between short-term and long-term patterns, and this results in lower accuracy when different nodes produce dependent log data. The existing detection methods fail to give post-hoc explanations, which makes it hard for the analysts to understand the causes of anomalies, their starting point, and the cause. The research is based on previous work by implementing the TCN-Transformer hybrid system, which will be able to find patterns at both short and long-time scales in distributed log data. The system employs PEL to enhance the working and quality of the preprocessing operations of the logs. The model adds three new elements, which include a post-hoc attribution module through SHAP and Integrated Gradients, a propagation analysis layer, and an overhead evaluation module. The
model will result in improving transparency and performance and will be able to cope with the issues as they arise. The designed (TLAN-EX) Temporal-Logical Attention Network with Explainability model will have the capability to achieve high accuracy and present clear explanations and low response time. The study revealed that the hybrid learning approach assists in enhancing multi-node anomaly detection and SHAP and Integrated Gradients help to justify the causes of anomalies. The propagation analysis is used to demonstrate the spread of system anomalies between nodes to indicate the detailed working pattern of the system. The system is analyzed to be at its optimal level as the model is highly functioning when the system is considered to be handling high traffic. The upgraded explanation capabilities and speed of detection of the model depict improved detection results and suit well in distributed financial systems.Temporal logical Model