Multi-Agent Reinforcement Learning for Intrusion Detection

This thesis presents a novel approach to provide adaptive mechanisms to detect and categorise Flooding-Base DoS (FBDoS) and Flooding-Base DDoS (FBDDoS) attacks. These attacks are generally based on a flood of packets with the intention of overfilling key resources of the target, and today the attacks have the capability to disrupt networks of almost any size. To address this problem we propose a Multi-Agent Reinforcement Learning (MARL) approach. In Reinforcement Learning (RL) agents learn to act optimally via observations and feedback from the environment in the form of positive or negative rewards. The thesis also investigates new methods of how to overcome some of the problems that Multi-Agent RL (MARL) faces. The proposed approach uses an architecture of distributed sensor and decision agents. Sensor agents extract network-state information. They receive only partial information about the global state of the environment and they map this local state to communication actions signals. Decision agents are located at a higher hierarchical level than sensor agents. Without any previous semantic knowledge about the signals, decision agents learn to interpret them and consequently interact with the environment. By means of this on-line process, sensor and decision agents learn the semantics of the communication action-signals. To expand our proposal to a large number of agents we deployed a hierarchical architecture composed of several levels. In this hierarchical architecture, communication signals flow from lower to higher hierarchical layers. To evaluate our architecture with large numbers of agents and a variety of information sources we used two simulated environments and created diverse tests emulating attacks under different network conditions. We found that our approach yielded positive results in its performance levels using predefined criteria. In the network environment we evaluated the performance of our proposal versus hand-coded solutions emulating simple misuse intrusion detection and a hybrid approach using misuse and anomaly methods. We found that our learning approach generates better results than the simple hand-coded misuse methods. Even though the hybrid hand-coded approach shows slightly better results than the learning mechanism, the main advantage of our learning method is that it does not need a designer with deep prior knowledge about the network environment. The agent architecture and the RL for signalling approach presented in this research can be applied to domains other than IDS. Domains where this methodology could be applied are Intrusion Prevention Systems, Network Management and Quality of Service enforcement.