logdata-anomaly-miner：一款安全日志解析与异常检测工具

官方公众号企业安全新浪微博

FreeBuf.COM网络安全行业门户，每日发布专业的安全资讯、技术剖析。

FreeBuf+小程序

FreeBuf+小程序把安全装进口袋

工具

关于logdata-anomaly-miner

logdata-anomaly-miner是一款安全日志解析与异常检测工具，该工具旨在以有限的资源和尽可能低的权限运行分析，以使其适合生产服务器使用。

为了确保 logdata-anomaly-miner的正常运行，推荐安装了python >= 3.6的Linux 系统。更具体地说，支持的系统包括 Debian Buster、Debian Bullseye、Debian Bookworm、Ubuntu 20.04、Ubuntu 22.04、Fedora（docker image fedora:latest）和 RedHat（docker image redhat/ubi9）。

工具架构

工具要求

scipy==1.10.0
pylibacl==0.5.4
kafka_python==2.0.2
pytz==2020.4
urllib3==1.26.19
numpy==1.22.0
Cerberus==1.3.2
psutil==5.7.3
kafka==1.3.5
pyzmq==20.0.0
python_dateutil==2.8.1
PyYAML==5.4
statsmodels==0.12.2

工具安装

由于该工具基于Python 3开发，因此我们首先需要在本地设备上安装并配置好最新版本的Python 3环境。

源码安装

接下来，广大研究人员可以直接使用下列命令将该项目源码克隆至本地：

git clone https://github.com/ait-aecid/logdata-anomaly-miner.git

然后切换到项目目录中，使用pip命令和项目提供的requirements.txt安装该工具所需的其他依赖组件：

cd logdata-anomaly-miner

pip install -r requirements.txt

Debian安装

在官方 Debian/Ubuntu 存储库中，有适用于 logdata-anomaly-miner 的 Debian 软件包：

apt-get update && apt-get install logdata-anomaly-miner

Wget安装

以下命令将安装最新的稳定版本：

cd $HOME

wget https://raw.githubusercontent.com/ait-aecid/logdata-anomaly-miner/main/scripts/aminer_install.sh

chmod +x aminer_install.sh

./aminer_install.sh

工具配置

现在让我们将 Apache 解析器模型添加到 aminer-config：

alice@ubuntu2004:~$ sudo ln -s /etc/aminer/conf-available/generic/ApacheAccessModel.py /etc/aminer/conf-enabled/

alice@ubuntu2004:~$

在以前版本的 aminer 中，我们必须用 python 编写配置文件。在当前版本中，我们可以使用以 yaml 编写的配置。现在创建并编辑文件 /etc/aminer/config.yml：

LearnMode: True

 

LogResourceList:

        - 'file:///var/log/apache2/access.log'

 

Parser:

        - id: 'START'

          start: True

          type: ApacheAccessModel

          name: 'apache'

 

Input:

        timestamp_paths: "/accesslog/time"

 

Analysis:

        - type: "NewMatchPathValueDetector"

          paths: ["/accesslog/status"]

          output_logline: True

 

EventHandlers:

        - id: "stpe"

          type: "StreamPrinterEventHandler"

如果我们现在启动 aminer，它将读取 access.log 并了解所有解析器路径。我们将在启动 aminer 之前使用“-C”参数清除持久性。（请注意，您可以使用 CTRL+c 终止 aminer）

alice@ubuntu2004:~$ sudo cat /var/log/apache2/access.log
127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"
alice@ubuntu2004:~$ sudo aminer -C --config /etc/aminer/config.yml
2021-05-17 12:12:36 New path(es) detected
NewMatchPathDetector: "DefaultNewMatchPathDetector" (1 lines)
/accesslog: 127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"
/accesslog/host: 127.0.0.1
/accesslog/sp0:
/accesslog/ident: -
/accesslog/sp1:
/accesslog/user: -
/accesslog/sp2:
/accesslog/time: 1621250714
/accesslog/sp3: ] "
/accesslog/fm/request: GET / HTTP/1.1
/accesslog/fm/request/method: 0
/accesslog/fm/request/sp5:
/accesslog/fm/request/request: /
/accesslog/fm/request/sp6:
/accesslog/fm/request/version: HTTP/1.1
/accesslog/sp6: "
/accesslog/status: 200
/accesslog/sp7:
/accesslog/size: 11229
/accesslog/combined: "-" "Wget/1.20.3 (linux-gnu)"
/accesslog/combined/combined: "-" "Wget/1.20.3 (linux-gnu)"
/accesslog/combined/combined/sp9: "
/accesslog/combined/combined/referer: -
/accesslog/combined/combined/sp10: " "
/accesslog/combined/combined/user_agent: Wget/1.20.3 (linux-gnu)
/accesslog/combined/combined/sp11: "
['/accesslog', '/accesslog/host', '/accesslog/sp0', '/accesslog/ident', '/accesslog/sp1', '/accesslog/user', '/accesslog/sp2', '/accesslog/time', '/accesslog/sp3', '/accesslog/fm/request', '/accesslog/sp6', '/accesslog/status', '/accesslog/sp7', '/accesslog/size', '/accesslog/combined', '/accesslog/combined/combined', '/accesslog/combined/combined/sp9', '/accesslog/combined/combined/referer', '/accesslog/combined/combined/sp10', '/accesslog/combined/combined/user_agent', '/accesslog/combined/combined/sp11', '/accesslog/fm/request/method', '/accesslog/fm/request/sp5', '/accesslog/fm/request/request', '/accesslog/fm/request/sp6', '/accesslog/fm/request/version']
127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"
2021-05-17 12:12:36 New value(s) detected
NewMatchPathValueDetector: "NewMatchPathValueDetector2" (1 lines)
{'/accesslog/status': 200}
127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"

工具运行演示

演示视频：【点我观看】

许可证协议

本项目的开发与发布遵循GPL-3.0开源许可协议。

项目地址

logdata-anomaly-miner：【GitHub传送门】

参考资料

https://dl.acm.org/doi/full/10.1145/3567675
http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0006643003860397
https://ieeexplore.ieee.org/document/8717887
https://securitylab.no/cyberhunt2019/

本文为独立观点，未经允许不得转载，授权请联系FreeBuf客服小蜜蜂，微信：freebee2022