LogNG: an Online Log Parsing Method Based on N-Gram

EasyChair Preprint 7444

Abstract

The first step in automatic log analysis is log parsing. The number of logs exploded with the increase in system size. Manual analysis of logs has become a difficult problem. To solve this problem, we proposed logNG, an online log parsing method based on N-gram that can efficiently parse logs in a streaming manner without the requirement for historical data training. When log messages are input in a stream, we first divide log messages of different lengths into different log groups. We’ll use an intuitive and simple assumption: If continuous multiple different tokens appear between log messages, these log messages belong to different log template. For each log group, we will use N-gram for template matching to further group log messages within our assumption. We evaluate and compare logNG with other log parsers on public data sets. The results of the experiments reveal that logNG can achieve the highest accuracy and efficiency.

Keyphrases: Log Parsing, n-gram, online algorithm

@booklet{EasyChair:7444,
  author    = {Xiangrui Liu and Shi Ying and Xinquan Ge and Shengkang Hu and Tiangang Li},
  title     = {LogNG: an Online Log Parsing Method Based on N-Gram},
  howpublished = {EasyChair Preprint 7444},
  year      = {EasyChair, 2022}}