通过Elasticsearch机器学习实现业务数据的智能检测和预测_检索分析服务 Elasticsearch版-阿里云帮助中心

Elasticsearch机器学习是一种利用机器学习技术对Elasticsearch数据进行智能检测和预测的工具，可以自动识别数据模式和数据异常，生成新的特征和聚合结果，为数据分析和应用提供支持。Elasticsearch机器学习可以提高数据的可用性和价值，还可以为用户提供更加智能和高效的数据分析和应用解决方案。本文通过两个示例介绍无监督机器学习和监督机器学习的实践。

背景信息

Elasticsearch机器学习分为无监督机器学习Unsupervised和监督机器学习Supervised两类：

无监督机器学习包括Single metric和Populartion等场景，对数据进行异常检测。该模式不需要训练机器学习什么是异常，机器学习算法将自动检测数据中的异常或异常模式。
监督机器学习包括Regression和Classification等场景，使用分类和回归算法解决非常复杂的问题。该模式需要一定的数据训练预测模型，然后使用训练出来的模型来对未来的数据进行分类、预测。

场景大类	场景	类别	说明
异常检测Anomaly Detection	单一指标检测Single metric	unsupervised	检测单个时序中的异常，数据分析仅在一个索引字段上执行。
	多指标检测Mutil metric	unsupervised	使用一个或多个指标检测异常，并根据需要拆分分析，数据分析在多个索引字段上执行。
	填充Populartion	unsupervised	通过与Population中的行为进行比较，检测不寻常的行为。 Population是指某一研究领域内所有可能被研究的个体、事物或现象的总体。
	高级用法和功能 Advanced	unsupervised	提供了更多的选项和设置，以便用户可以更好地定制和优化机器学习模型，以适应不同的应用场景和数据类型，用于更高级用例的机器学习。
	归类Categorization	unsupervised	识别和分析日志消息中的特征和模式，将日志消息分为不同的组别，并检测其中的异常情况。
数据分析Data Frame Analytics	离群值检测Outlier detection	unsupervised	聚类分析和异常检测算法来训练模型，用于快速检测数据中的异常点或异常行为。
	回归Regression	supervised	回归预测数据集中的数值。
	分类Classification	supervised	分类预测数据集中数据点的类别。

准备工作

创建阿里云Elasticsearch实例，本文使用8.5版本Elasticsearch实例。具体操作，请参见创建阿里云Elasticsearch实例。

{
  "_index": "kibana_sample_data_logs",
  "_type": "_doc",
  "_id": "n6GHI4gBmNQSVxOwNnPn",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24",
    "bytes": 847,
    "clientip": "122.62.233.59",
    "extension": "",
    "geo": {
      "srcdest": "CN:CO",
      "src": "CN",
      "dest": "CO",
      "coordinates": {
        "lat": 31.24905556,
        "lon": -82.39530556
    "host": "www.elastic.co",
    "index": "kibana_sample_data_logs",
    "ip": "122.62.233.59",
    "machine": {
      "ram": 4294967296,
      "os": "win xp"
    "memory": null,
    "message": "122.62.233.59 - - [2018-08-21T02:34:54.901Z] \"GET /logging HTTP/1.1\" 200 847 \"-\" \"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24\"",
    "phpmemory": null,
    "referer": "http://twitter.com/success/paul-w-richards",
    "request": "/logging",
    "response": 200,
    "tags": [
      "success",
      "info"
    "timestamp": "2023-06-06T02:34:54.901Z",
    "url": "https://www.elastic.co/solutions/logging",
    "utc_time": "2023-06-06T02:34:54.901Z",
    "event": {
      "dataset": "sample_web_logs"
  "fields": {
    "@timestamp": [
      "2023-06-06T02:34:54.901Z"
    "utc_time": [
      "2023-06-06T02:34:54.901Z"
    "hour_of_day": [
    "timestamp": [
      "2023-06-06T02:34:54.901Z"
  "sort": [
    1686018894901
}

{
  "_index": "kibana_sample_data_flights",
  "_type": "_doc",
  "_id": "7b0aeogBmNQSVxOwslB_",
  "_version": 1,
  "_score": null,
  "_source": {
    "FlightNum": "QYX9S3I",
    "DestCountry": "CH",
    "OriginWeather": "Cloudy",
    "OriginCityName": "Chicago",
    "AvgTicketPrice": 824.8516378170061,
    "DistanceMiles": 4442.909325899777,
    "FlightDelay": false,
    "DestWeather": "Thunder & Lightning",
    "Dest": "Zurich Airport",
    "FlightDelayType": "No Delay",
    "OriginCountry": "US",
    "dayOfWeek": 4,
    "DistanceKilometers": 7150.1694661808515,
    "timestamp": "2023-06-02T07:28:15",
    "DestLocation": {
      "lat": "47.464699",
      "lon": "8.54917"
    "DestAirportID": "ZRH",
    "Carrier": "Logstash Airways",
    "Cancelled": false,
    "FlightTimeMin": 420.59820389299125,
    "Origin": "Chicago O'Hare International Airport",
    "OriginLocation": {
      "lat": "41.97859955",
      "lon": "-87.90480042"
    "DestRegion": "CH-ZH",
    "OriginAirportID": "ORD",
    "OriginRegion": "US-IL",
    "DestCityName": "Zurich",
    "FlightTimeHour": 7.009970064883188,
    "FlightDelayMin": 0
}

```
GET _ml/inference/flightdelaymin_job*?human=true
```

PUT _ingest/pipeline/flight_flightDelayMin_predict
  "description": "Predict the number of minutes of delay for each flight",
  "processors": [
      "inference": {
        "model_id": "flightDelayMin_job-168609891****",
        "inference_config": {
          "regression": {}
        "field_map": {},
        "tag": "flightDelayMin_prediction"
}

POST _ingest/pipeline/flight_flightDelayMin_predict/_simulate
  "docs": [
      "_source": {
        "FlightNum": "EDGSV3T",
        "DestCountry": "CN",
        "OriginWeather": "Damaging Wind",
        "OriginCityName": "Durban",
        "AvgTicketPrice": 1065.7037805199147,
        "DistanceMiles": 7273.460817641552,
        "FlightDelay": true,
        "DestWeather": "Rain",
        "Dest": "Shanghai Pudong International Airport",
        "FlightDelayType": "Carrier Delay",
        "OriginCountry": "ZA",
        "dayOfWeek": 5,
        "DistanceKilometers": 11705.500526106527,
        "timestamp": "2023-06-03T09:34:00",
        "DestLocation": {
          "lat": "31.14340019",
          "lon": "121.8050003"
        "DestAirportID": "PVG",
        "Carrier": "Kibana Airlines",
        "Cancelled": false,
        "FlightTimeMin": 881.1071804361806,
        "Origin": "King Shaka International Airport",
        "OriginLocation": {
          "lat": "-29.61444444",
          "lon": "31.11972222"
        "DestRegion": "SE-BD",
        "OriginAirportID": "DUR",
        "OriginRegion": "SE-BD",
        "DestCityName": "Shanghai",
        "FlightTimeHour": 14.685119673936343,
        "FlightDelayMin": 45
}

背景 信息

准备工作

创建机器学习模型

创建单指标机器学习模型

创建推理机器学习模型

训练航班延误预测模型

使用航班延误预测模型

背景信息