Vector 调研测试同步至 MinIO - 小红同学的博客

Hwsdien included in BigData

2023-09-15 2023-09-15 333 words 2 minutes

Contents

前言

继续接着《Vector 调研测试之环境安装》，本文旨在记录对 Vector 写入 MinIO 的功能性测试是否能够满足广泛的需求，以期能够为他人提供有用的参考和帮助。

准确工作

新建 Kafka Topic

1

kafka-topics -–create -–bootstrap-server kafka-1:9092 -–replication-factor 1 -–partitions 6 -–topic bigdata-data-v1

查看 Kafka Topic

1

kafka-topics --list --bootstrap-server kafka-1:9092

测试数据

测试数据来源

https://www.kaggle.com/datasets/eliasdabbas/web-server-access-logs

测试数据Demo

1
2
3
4
5
6


54.36.149.41 - - [22/Jan/2019:03:56:14 +0330] "GET /filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53 HTTP/1.1" 200 30577 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)" "-" 
31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/60844/productModel/200x200 HTTP/1.1" 200 5667 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-" 
31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/61474/productModel/200x200 HTTP/1.1" 200 5379 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-" 
40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] "GET /image/14925/productModel/100x100 HTTP/1.1" 200 1696 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-" 
91.99.72.15 - - [22/Jan/2019:03:56:17 +0330] "GET /product/31893/62100/%D8%B3%D8%B4%D9%88%D8%A7%D8%B1-%D8%AE%D8%A7%D9%86%DA%AF%DB%8C-%D9%BE%D8%B1%D9%86%D8%B3%D9%84%DB%8C-%D9%85%D8%AF%D9%84-PR257AT HTTP/1.1" 200 41483 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0)Gecko/16.0 Firefox/16.0" "-" 
40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] "GET /image/23488/productModel/150x150 HTTP/1.1" 200 2654 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"

写入数据

1

kafka-console-producer --broker-list kafka-1:9092 --topic bigdata-data-v1 < ./access.log

消费数据查看

1

kafka-console-consumer --bootstrap-server kakfa-1:9092 --topic bigdata-data-v1 --from-beginning

MinIO 设置

创建 Bucket
创建 AK 信息

Access Key: ofbHkAZT0CEsIgq4muR8

Secret Key: 6T9ap5ywJfxad0oswUFVx0M3AdSCA5fv0cvH3Daq

写入 MinIO

测试从Kafka写入到MinIO

vim /opt/vector/config/minio.toml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


[api]
enabled = true
address = "0.0.0.0:8686"

[sources.src_kafka]
type = "kafka"
bootstrap_servers = "192.168.31.37:9092"
auto_offset_reset = "beginning"
group_id = "consumer-bigdata-es"
topics = ["bigdata-data-v1"]
decoding.codec = "bytes"

[transforms.transform_log]
type = "remap"
inputs = ["src_kafka"]
source = """
. = parse_nginx_log!(string!(.message), "combined")
.date, err = format_timestamp(.timestamp, "%Y-%m-%d")
"""

[sinks.sink_minio]
type = "aws_s3"
inputs = ["transform_log"]
region = "home"
bucket = "log-bucket"
auth.access_key_id = "ofbHkAZT0CEsIgq4muR8"
auth.secret_access_key = "6T9ap5ywJfxad0oswUFVx0M3AdSCA5fv0cvH3Daq"
encoding.codec = "native_json"
compression = "none"
endpoint = "http://192.168.31.37:9000"
key_prefix = "access-log/{{ date }}/"
filename_extension = "json"
filename_time_format = "%s"
batch.max_bytes = 100000000
batch.timeout_secs = 60

运行

1

/opt/vector/bin/vector -c /opt/vector/config/minio.toml

查看文件是否生成
抽查文件内容

结论

经过部署测试，Vector 用 VRL 解析转换再写入 MinIO 在功能上是完全OK的。