Matt Jansen, Rakesh Bobba, Dave Nevin (Oregon State University)

Provenance-based Intrusion Detection Systems (PIDS) are threat detection methods which utilize system provenance graphs as a medium for performing detection, as opposed to conventional log analysis and correlation techniques. Prior works have explored the creation of system provenance graphs from audit data, graph summarization and indexing techniques, as well as methods for utilizing graphs to perform attack detection and investigation. However, insufficient focus has been placed on the practical usage of PIDS for detection, from the perspective of end-user security analysts and detection engineers within a Security Operations Center (SOC). Specifically, for rule-based PIDS which depend on an underlying signature database of system provenance graphs representing attack behavior, prior work has not explored the creation process of these graph-based signatures or rules. In this work, we perform a user study to compare the difficulty associated with creating graph-based detection, as opposed to conventional log-based detection rules. Participants in the user study create both log and graph-based detection rules for attack scenarios of varying difficulty, and provide feedback of their usage experience after the scenarios have concluded. Through qualitative analysis we identify and explain various trends in both rule length and rule creation time. We additionally run the produced detection rules against the attacks described in the scenarios using open source tooling to compare the accuracy of the rules produced by the study participants. We observed that both log and graph-based methods resulted in high detection accuracy, while the graph-based creation process resulted in higher interpretability and low false positives as compared to log-based methods.