目的在 logstash 中自定义一个 customer_time 字段,获取日志的时间,并且替换 @timestamp 中的值
ELK 的架构为 filebeat > redis > Logstash > Elastash > Kibana
日志为:
22-Jun-2018 17:45:22.397 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.0.26
22-Jun-2018 17:45:22.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Aug 18 2015 11:38:37 UTC
22-Jun-2018 17:45:22.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.0.26.0
22-Jun-2018 17:45:22.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
22-Jun-2018 17:45:22.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 3.10.0-862.2.3.el7.x86_64
配置文件 logstash.conf 内容为:
input {
redis {
port => "6379"
host => "elk_redis"
data_type => "list"
key => "bossmobile"
password => "ibalife"
}
}
filter {
grok {
match => [ "message" , "(?<customer_time>%{MONTHDAY}\-%{MONTH}\-%{YEAR}\s+%{TIME})" ]
}
date {
match => [ "customer_time", "YYYY-MM-dd;HH:mm:ss.SSS", "ISO8601" ]
locale => "en"
target => [ "@timestamp" ]
timezone => "Asia/Shanghai"
}
}
output {
if [fields][service] == "bossmobile_catalina" {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "bossmobile_catalina-%{+YYYY.MM.dd}"
}
}
if [fields][service] == "bossmobile_ibalife" {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "bossmobile_ibalife-%{+YYYY.MM.dd}"
}
}
}
获取其中的一个json 文件内容为
{
"_index": "bossmobile_catalina-2018.07.11",
"_type": "doc",
"_id": "39KniGQBujdRsWjAfuSP",
"_version": 1,
"_score": null,
"_source": {
"message": "22-Jun-2018 17:45:22.400 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms4096M",
"tags": [
"_dateparsefailure"
],
"input": {
"type": "log"
},
"prospector": {
"type": "log"
},
"source": "/iba/ibaboss/java/bossmobile-tomcat-8.0.26/logs/catalina.out",
"@timestamp": "2018-07-11T09:22:40.889Z",
"@version": "1",
"beat": {
"name": "localhost.localdomain",
"hostname": "localhost.localdomain",
"version": "6.3.0"
},
"fields": {
"service": "bossmobile_catalina"
},
"offset": 2145,
"host": {
"name": "localhost.localdomain"
},
"customer_time": "22-Jun-2018 17:45:22.400"
},
"fields": {
"@timestamp": [
"2018-07-11T09:22:40.889Z"
]
},
"sort": [
1531300960889
]
}
从 JSON 文件中看出,customer_time 已经成功获取了日志中的时间,但是没有替换 @timestamp 中的值。google 了很久也没找到答案,不知道各位有没有遇到过相似的问题。谢谢~
match => [ "customer_time", "YYYY-MM-dd;HH:mm:ss.SSS", "ISO8601" ] 这里写得有问题,customer_time 是 22-Jun-2018 17:45:22.400 ,格式化的格式应为 dd-MMM-yyyy HH:mm:ss.SSS