apache nifi - RouteOnAttribute based on list

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

In my NiFi pipeline I have some flow files that ran into an issue with a Python script running on the ExecuteStreamCommand processor. When they fail, they come out as 0 byte flow files so I can't look and see what might be causing the issue nor how to fix it. Luckily, the flow file is not just gone forever: it exists in S3 with about 60 million other files. However, I do not want to mass re-pull from S3 and have to manually comb through to find each file that filed.

Instead, what I've concocted is that I can pull a specific id that's in the attributes of the failed, empty flow files and throw it into a list thanks to AttributetoJSON . What I would like to do is then re-pull from S3 and run those through a RouteOnAttribute processor that will keep flow files whose id appears in the list, and then discard those that don't. However, I'm not seeing a clear way to use the list in my RouteOnAttribute processor. Is there a way to do something like ${nameid} in [123, 345, 567, 789] ?

@daggett I'm not quite sure what you mean? when it comes out of the ExecuteStreamCommand as non-zero exit status it comes out as a 0 byte file. – carousallie Nov 13, 2019 at 21:23 Sorry, I mixed ExecuteScript and ExecuteCommand. To match multiple values, you could use match function like this:


    ${myattr:matches('^(123|345|678)$')}

– daggett Nov 14, 2019 at 6:07 @daggett Gotcha, was wondering if I had seriously missed something major. Thank you, that looks perfect! – carousallie Nov 14, 2019 at 15:47

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question . Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers .