Ask Your Question

Moving all root level fields to another field?

asked 2019-03-01 08:30:25 -0500

daveh gravatar image

I have a SDC RPC pipeline that receives records from many other pipelines upon failure in the other pipelines so we can centralize our error processing. The SDC RPC pipeline needs to create a Kafka message that ElasticSearch's Logstash can consume to post the error and information to ElasticSearch.

The need is to initially be able to move all of the top level fields into another field -- /alert_detais -- without knowing in advance what the top level fields are (the can vary over all the other pipelines that will send their pipeline errors to this SDC RPC pipeline).

Looking for suggestions to move a root level field (e.g., /elapsed) to a 'child field' (/alert_details/elapsed). Again the root level fields are not know and of varying count when this SDC RPC pipeline gets the record from the calling pipeline.

Also, tying to see if I can avoid a jython processor this.


edit retag flag offensive close merge delete

2 Answers

Sort by » oldest newest most voted

answered 2019-03-01 15:44:29 -0500

iamontheinet gravatar image


Given your example, you could use Expression Evaluator, like so:

image description

Here's the resulting output:

image description

Cheers, Dash

edit flag offensive delete link more


Hello Dash, thanks for replying, however I know that if the root level fields are known in advance one can create a "hard coded" processor, which I have already done. This challenge is what if the root level fields are NOT known in advanced? Regards.

daveh gravatar imagedaveh ( 2019-03-04 08:26:18 -0500 )edit

You mean if root level field '/elapsed' is not known? Then you can use expression ${record:value('/')} which will move all root level fields to a given child field.

iamontheinet gravatar imageiamontheinet ( 2019-03-04 09:11:37 -0500 )edit

Hmmm seems that, for me, this suggestion copies the root fields to the child but does not move them. Using an Expression Evaluator: Output Field: /x Field Expression: ${record:value('/')} Thoughts? And Thanks.

daveh gravatar imagedaveh ( 2019-03-05 12:39:21 -0500 )edit

Yes i must admit that they are. Was exploring different options

daveh gravatar imagedaveh ( 2019-03-05 13:19:08 -0500 )edit

answered 2019-03-05 13:01:25 -0500

metadaddy gravatar image

If you want to stick with the off-the-shelf processors, you can do as @iamontheinet suggests, and add a Field Remover processor configured to keep only the `/alert_details' field. Alternatively, you can use a script evaluator to restructure the record.

Here's an example in Jython:

for record in records:
    # Put old root in a new map, under /alert_details
    record.value = {'alert_details' : record.value}    

    # Write record to processor output

  except Exception as e:
    # Send record to error
    error.write(record, str(e))

Note that using off-the-shelf processors is almost always much faster than reaching for a script evaluator, but some things are easy to do in a script that are much more convoluted without a few lines of code.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-03-01 08:30:25 -0500

Seen: 423 times

Last updated: Mar 05 '19