如何通过Logstash实现多字段数据整合_检索分析服务 Elasticsearch版(ES)-阿里云帮助中心

在使用Logstash传输数据时，如果您需要通过合并字段来整合数据，例如将a字段和b字段合并为一个新的c字段，然后移除a字段和b字段，可以通过logstash-filter-mutate插件的多个模块实现。此插件为系统默认安装插件，无须再安装，且不支持卸载。本文介绍如何通过logstash-filter-mutate插件实现多字段合并。

背景信息

logstash-filter-mutate插件支持对事件中的字段进行重命名、删除、替换和修改操作。配置文件中的mutate按照下表中的顺序执行，详细信息请参见 Mutate filter plugin 。

模块	输入类型
coerce	hash
rename	hash
update	hash
replace	hash
convert	hash
gsub	array
uppercase	array
capitalize	array
lowercase	array
strip	array
remove_field	array
split	hash
join	hash
merge	hash
copy	hash

前提条件

创建阿里云Elasticsearch实例。

具体操作，请参见创建阿里云Elasticsearch实例，本文以7.10版本实例为例。
开启目标Elasticsearch实例的自动创建索引功能。

具体操作请参见配置YML参数。

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    "max_score" : 1.0,
    "hits" : [
        "_index" : "yc_text",
        "_type" : "_doc",
        "_id" : "HpIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "app.name" : "app1",
          "annual_rate" : "31%",
          "describe" : "可以自助选择消息推送",
          "message" : "10000"
        "_index" : "yc_text",
        "_type" : "_doc",
        "_id" : "H5IduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "app.name" : "app2",
          "annual_rate" : "35%",
          "describe" : "每天收益到账消息推送",
          "message" : "10001"
        "_index" : "yc_text",
        "_type" : "_doc",
        "_id" : "IpIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "app.name" : "app3",
          "annual rate" : "30",
          "describe" : "每天收益会消息推送",
          "message" : "10004"
        "_index" : "yc_text",
        "_type" : "_doc",
        "_id" : "IJIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "app.name" : "app4",
          "annual_rate" : "38%",
          "describe" : "每天收益立即到账消息推送",
          "message" : "10002"
        "_index" : "yc_text",
        "_type" : "_doc",
        "_id" : "IZIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "app.name" : "app5",
          "annual_rate" : "40%",
          "describe" : "每天收益到账消息推送",
          "message" : "10003"
        "_index" : "yc_text",
        "_type" : "_doc",
        "_id" : "I5IduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "app.name" : "app6",
          "annual_rate" : "33%",
          "describe" : "通过短信提示获取收益消息",
          "message" : "10005"
}

```
input {
    elasticsearch {
        hosts => ["http://es-cn-tl3264bqv001d****.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "yc_text"
        docinfo => true
filter {
    mutate {
        merge =>  { "message" => "app.name" }
    mutate {
        rename => [ "message","anger" ]
    mutate {
        remove_field => [ "app.name" ]
output {
    elasticsearch {
        hosts => ["http://es-cn-tl3264bqv001d****.elasticsearch.aliyuncs.com:9200"]
        user => "elastic"
        password => "your_password"
        index => "yc_text_new"
        document_type => "%{[@metadata][_type]}"
        document_id => "%{[@metadata][_id]}"
                        
```

GET yc_text_new/_search
  "query": {
    "match_all": {}
}

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    "max_score" : 1.0,
    "hits" : [
        "_index" : "yc_text_new",
        "_type" : "_doc",
        "_id" : "H5IduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "describe" : "每天收益到账消息推送",
          "@version" : "1",
          "anger" : [
            "10001",
            "app2"
          "@timestamp" : "2021-12-15T03:45:25.321Z",
          "annual_rate" : "35%"
        "_index" : "yc_text_new",
        "_type" : "_doc",
        "_id" : "IZIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "describe" : "每天收益到账消息推送",
          "@version" : "1",
          "anger" : [
            "10003",
            "app5"
          "@timestamp" : "2021-12-15T03:45:25.321Z",
          "annual_rate" : "40%"
        "_index" : "yc_text_new",
        "_type" : "_doc",
        "_id" : "I5IduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "describe" : "通过短信提示获取收益消息",
          "@version" : "1",
          "anger" : [
            "10005",
            "app6"
          "@timestamp" : "2021-12-15T03:45:25.322Z",
          "annual_rate" : "33%"
        "_index" : "yc_text_new",
        "_type" : "_doc",
        "_id" : "HpIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "describe" : "可以自助选择消息推送",
          "@version" : "1",
          "anger" : [
            "10000",
            "app1"
          "@timestamp" : "2021-12-15T03:45:25.298Z",
          "annual_rate" : "31%"
        "_index" : "yc_text_new",
        "_type" : "_doc",
        "_id" : "IJIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "describe" : "每天收益立即到账消息推送",
          "@version" : "1",
          "anger" : [
            "10002",
            "app4"
          "@timestamp" : "2021-12-15T03:45:25.321Z",
          "annual_rate" : "38%"
        "_index" : "yc_text_new",
        "_type" : "_doc",
        "_id" : "IpIduH0BWiRrY8Azn65i",
        "_score" : 1.0,
        "_source" : {
          "describe" : "每天收益会消息推送",
          "@version" : "1",
          "anger" : [
            "10004",
            "app3"
          "@timestamp" : "2021-12-15T03:45:25.321Z",
          "annual rate" : "30"
}

背景信息

前提条件

操作步骤

验证结果