Adding Kalman filter into ESI: an anomaly detection approach for middleware metadata

Abstract

This report treats about the implementation of the Kalman filter algorithm into a product of Border Innovation ‐ ESI – which had the goal to complement the range of algorithms available in the tool. Most specifically it treats about using the referred algorithm to point out anomalous events on a very particular Big‐data scenario, which is the metadata about the data flowing in streaming through a middleware software. The focus of this report thus relies on providing reasoning about the problem which the algorithm has to deal with, and the data treatment adopted to implement the algorithm into the referred tool. It is relevant to mention that the practical implementation was done using Splunk, and therefore the technological aspects and the language of the tool were important factors that guided the algorithm implementation. The project took place during the author’s work at the company and the data used to guide the project reflects the reality the tool is built to deal with, and it is fully anonymized to preserve the company interest.