[WakaWaka]

Does anyone know of a good way to pass in extra data to each call of the map function in a Map/Reduce script? For example, lets say I some data from a custom record that I need within each call to the map function. It would be silly to have to load or look up the fields of the custom record every single time, right?

I know that I can make it work so that it only has to be loaded once per queue, but how could the information in the custom record be shared across multiple queues and only load it once? Loading a record a few more times isn’t a big deal, but I can think of other things where lots of governance could be saved if there was a way to share data across multiple queues.

I was thinking of using the Cache module, but will it error out if two queues try to read a value at once? For that matter, the documentation talks about the Cache being accessible in different scripts, but i have no idea how it would work across different queues of the same script.

I would try some of this stuff out myself, but I am currently waiting around until I have access to an account with multiple queues. In the meantime, I figured I’d ask around on here.

Thanks in advance.
This is a cached copy. Click here to see the original post.

[david.smith]

cache module works well for me.

[MChammaTX]

What about in the GetInput stage? You can set an object property and store it there. If you do it carefully you might even be able to only do 1 query and match to associate the data.

[Author]

Posts