Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I'm attempting to simplify my logstash config. I want to split the program field into separate fields (as show below) however I would prefer to use just one grok statement (if it's at all possible!)

Of the two examples below I get an _grokparsefailure on the second example, but not the first. Since grok has the add_field and remove_field options I would assume that I could combine it all into one grok statement. Why is this not the case? Have I missed some ordering/syntax somewhere?

Sample log:

2016-02-16T16:42:06Z ubuntu docker/THISTESTNAME[892]: 172.16.229.1 - - [16/Feb/2016:16:42:06 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36" "-"
Why does this work:
filter {
       # Extracts the docker name, ID, image etc elements
       mutate {
               add_field => { "[@metadata][program]" => "%{program}" }
               remove_field => "[program]"
       grok {
                patterns_dir => "/logstash/patterns_dir/docker"
                match => { "[@metadata][program]" => "%{D_ID}" }
But this does not:
filter {
        grok {
               add_field => { "[@metadata][program]" => "%{program}" }
               remove_field => "[program]"
               patterns_dir => "/logstash/patterns_dir/docker"
               match => { "[@metadata][program]" => "%{D_ID}" }
                Huh, so in a way if I insisted on doing it that way and only using grok the way it could logically be done would be (even though the first gork returns an error as there were no matches):          grok {                add_field => { "[@metadata][program]" => "%{program}" }                remove_field => "[program]"         }  ////        grok {                patterns_dir => "/logstash/patterns_dir/docker"                match => { "[@metadata][program]" => "%{D_ID}" }         }
– geekscrap
                Feb 16, 2016 at 17:11
                Using mutate to add the field and then using grok is fine, and is a better understood syntax than using grok to just run the add_field and remove_field.
– Alain Collins
                Feb 16, 2016 at 17:31
This was directly answered by @Alan, however I found this way a little more readable and compressed my code even more:
grok {
    patterns_dir => "/logstash/patterns_dir/docker-patterns"
    match => { "program" => "%{D_ID}" }
    overwrite => [ "program" ]
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.