[CREECE]

I’m writing a map/reduce script that relies on both script parameters as well as some other preferences specific to the custom application. I don’t want to create/run a search each time data is being processed so I thought I would use the ‘N/cache’ module. I then saw the ‘N/runtime’ session module as well. Both of these have storage options. Is ‘N/cache’ the more appropriate one to use for map/reduce? Is there a general rule to which one should with which script types?
This is a cached copy. Click here to see the original post.

[erictgrubaugh]

Can you explain a little more about “each time data is being processed”? Typically in a M/R script, you want to use the `getInputData` phase to retrieve everything you’ll need for processing your data as well as the data itself. Then you’d package all that knowledge up in a data structure that gets passed on to `map` and/or `reduce` phases directly. There should be nothing for those phases to go search or gather as everything is in the data you’ve already collected in `getInputData`.

That’s all in theory, of course. In practice there are many more considerations, so perhaps some more detail on your plan and usage of these params/prefs would help devise a better approach.

[david.smith]

In your case I would use the cache to store your data. You will need the runtime to get the parameters from the deployment and is typically used for user session information.

[CREECE]

erictgrubaugh I am caching my values in the getInput phase. Each phase is called once either per search result (map) or mapped data (reduce). If i need “preference values” or “script parameters” in these phases, then I don’t want to keep hitting the runtime or doing a search for the values. Therefore, i cache my values in my getInput phase and call the cache whenever i need the data in those other phases. I just wasn’t sure if that is the most appropriate usage as there is also the session storage that you can get/set data on as well.

[david.smith]

If it’s a large search I would cache it. However, if you’re just getting the script parameters I would use runtime. Whether you’re getting the values from runtime or cache I don’t think there would be much of a performance difference.

[CREECE]

david.smith I definitely have the large search cached. I used runtime to get my script parameters but cached them just to be on the safe side i guess. I don’t see any slowdown or speedup to be honest but i’m only running on ~800 records. I don’t have any good metrics to give on it. Summarize is still reporting roughly the same time either way.

As a side note, I’ve also noticed that the cache doesn’t appear to work as expected. I have a method that retrieves all of my values I want to cache and then a user defined method that takes that and caches the values (key->value) with the same cache name. When I retrieve a key, the loader is called whether or not the value exists if the loader is specified. The documentation states that the loader is called IFF the key is not found which is not what is happening. I took the loader off as I really don’t need it in my case since I am caching everything at once and everything works just fine.

[Author]

Posts