Explainable Deep Learning Approach: Interpretation Between Detection and Root Cause Analysis of Distributed Log Anomalies with TLAN-EX

Abstract

The deeper analysis into modern distributed banking environments has discovered that nowadays distributed log is facing challenges. This is because transaction data comes from multiple networked nodes, which makes a complete system behavior check difficult. This shows how complications grow when multiple system components produce logs at the same time. This gives rise to data inconsistencies that cause anomaly detection to be more difficult. The detection systems used nowadays show lower reliability due to high log volumes and logs that do not follow the standard format, and events showing no clear order, due to which the system raises alarms and fails to catch the real anomalies. Research into log anomaly detection by using Deep Learning (DL) models has shown positive results. However, distributed systems make it hard and difficult for real-life implementation. The current sequence models used in anomaly detection systems only identify one pattern type at a time between short-term and long-term patterns. This results in lower accuracy when different nodes produce dependent log data. The existing detection methods fail to give post-hoc explanations. This makes it hard for the analysts to understand the causes of anomalies, their starting point, and the cause. The current research is based on previous researches. The study implemented the TCN-Transformer hybrid system, which would be able to find patterns at both short and long-time scales in distributed log data. The system employs Parameter Entity Labeling (PEL) to enhance the working and quality of the preprocessing operations of the logs. The model adds three new elements, which include a post-hoc attribution module through SHAP and Integrated Gradients, a propagation analysis layer, and an overhead evaluation module. The model would result in improving the transparency and performance and might be able to cope with the issues as they arise. The designed (TLAN-EX) Temporal-Logical Attention Network with Explainability model has the capability to achieve high accuracy and present clear explanations and low response time. The study revealed that the hybrid learning approach assists in enhancing multi-node anomaly detection and SHAP and Integrated Gradients help to justify the causes of anomalies. The propagation analysis is used to demonstrate the spread of system anomalies between nodes to indicate the detailed working pattern of the system. The system is analyzed to be at its optimal level as the model is highly functioning when the system is considered to be handling high traffic. The upgraded explanation capabilities and speed of detection of the model depict improved detection results and suit well in distributed financial systems.