System Event Mining: Algorithms and Applications   (Tutorial Slides)


Generic placeholder image

Tao Li

Professor, Florida International University

View details »

Generic placeholder image

Larisa Shwartz

Researcher, IBM T.J. Watson Research Center, USA

View details »

Generic placeholder image

Genady Ya. Grabarnik

Professor, St. John's University, USA

View details »


Abstract

Many systems, from computing systems, physical systems, business systems, to social systems, are only observable indirectly from the events they emit. Events can be defined as real-world occurrences and they typically involve changes of system states. Events are naturally temporal and are often stored as logs, e.g., business transaction logs, stock trading logs, sensor logs, computer system logs, HTTP requests, database queries, network traffic data, etc. These events capture system states and activities over time. For effective system management, a system needs to automatically monitor, characterize, and understand its behavior and dynamics, mine events to uncover useful patterns, and acquire the needed knowledge from historical log/event data.

Event mining is a series of techniques for automatically and efficiently extracting valuable knowledge from historical event/log data and plays an important role in system management. The purpose of this tutorial is to present a variety of event mining approaches and applications with a focus on computing system management. It is mainly intended for researchers, practitioners, and graduate students who are interested in learning about the state of the art in event mining.

Audience

Students or practitioners with general understanding of data mining would have no problem following the tutorial.

Outline

  • Prelude

    • Problem Introduction
    • Overall landscape of Solutions
  • Part 1: Event Generation and System Monitoring

    • Event Generation: From Logs to Event
      • Log Parser
      • Log Message Classification
      • Log Message Clustering
    • Optimizing System Monitoring Configurations
      • Automatic Monitoring
      • Eliminating False Positives
      • Eliminating False Negatives
  • Part 2: Pattern Discovery and Summarization

    • Event Pattern Mining
      • Sequential Pattern/Frequent Episode
      • Fully Dependent Pattern/Mutually Dependent Pattern/T-Pattern
      • Periodic Pattern
      • Event Burst/Rare Event
      • Correlated Pattern between Time Series and Event
    • Mining Time Lags
      • Non-parametric Model
      • Parametric Model
    • Log Event Summarization
      • Summarizing with Frequency Change
      • Summarizing with Temporal Dynamics
      • Facilitating Summarization Tasks
  • Part 3: Mining with Events and Tickets

    • Ticket Classification
    • Ticket Resolutions Recommendation
  • Part 4: Applications

    • System Diagnosis
    • Resolution Automation
    • Root Cause Analysis

Duration and Session

2 hours (i.e., 1 hour, coffee break, 1 hour).


Tutors

Tao Li earned his PhD in computer science from the Department of Computer Science, the University of Rochester in 2004. He is currently a professor in the School of Computing and Information Sciences at Florida International University (FIU). His research interests are in data mining, information retrieval, and computing system management. He is a recipient of the NSF CAREER Award (2006-2010) and multiple IBM Faculty Research Awards. In 2009, he received the FIU Excellence in Research and Creativities Award, and in 2010, he received an IBM Scalable Data Analytics Innovation Award. He received the inaugural Mentorship Award from the College of Engineering and Computing at FIU in 2011 and the Excellence in Research Award from the College of Engineering and Computing at FIU in 2012. He is currently on the editorial board of ACM Transactions on Knowledge Discovery from Data (ACM TKDD), IEEE Transactions on Knowledge and Data Engineering (IEEE TKDE), and Knowledge and Information System Journal (KAIS). Dr. Li has published more than 30 papers in leading venues in the area of computing system management.

Larisa Shwartz is a senior technical researcher (STSM) at IBM T. J. Watson Research Center with research experience in mathematics and computer science. She received her Ph.D. degree in mathematics from UNISA University. Dr. Shwartz is focusing on analytics for IT service management, cloud systems and automation. She has over 50 publications and 51 patents. Dr. Shwartz is on editorial board of International Journal of Analytics and TPC member of conferences like ICSOC, SIMUL, SOCA. She is a recipient of IBM Corporate Award 2017 and number of IBM research awards, including outstanding technical achievement award in 2016 and outstanding technical innovation award in 2015.

Genady Ya. Grabarnik teaches in the Math and CS Department, St John’s University. He is a trained mathematician and has authored over 80 papers. He spent 10 years at IBM T.J.Watson Research Center where his work was celebrated with a number of awards, including Outstanding Technical Achievement Award and Research Achievement Awards. He is a prolific inventor with over 50 US patents. His interests include research in functional analysis, inventions, and research in computer science and artificial intelligence.

Contributors


Generic placeholder image

Liang Tang

View details »

Generic placeholder image

Chunqiu Zeng

View details »

Generic placeholder image

Yexi Jiang

View details »

Generic placeholder image

Chao Shen

View details »

Generic placeholder image

Wubai Zhou

View details »

Generic placeholder image

Qing Wang

View details »


Book and Survey

Important References