[darrenhillconsulting]

Ok,

So, I’m trying to spread some heavy duty processing for incoming data across lots of Map/Reduce deployments. My question is, how can I find out which deployment is currently ‘idle’? Further, reading the documentation, it appears I can ‘submit’ against the same deployment over and over, and achieve the same thing (I’ll get a null, from the submit)? Is that true?

I guess lastly, any alternative approaches to what I’m trying to do?
This is a cached copy. Click here to see the original post.

[MChammaTX]

Not sure if this helps but some heavy duty scripts I have use multiple queues. If a user tries to kick off another MR script and all the queues are blocked I display a message saying to try back in a few minutes as another process is running. I achieve this as I can check which scripts are currently running using a Scheduled Script Instance saved search which also luckily includes map reduce scripts.

[darrenhillconsulting]

Ok, thanks Maher! Step in the right direction for sure.

Now, I’ve been staring at this for a while. I’ve got 10 deployments for a Map/Reduce.

I want to run a Scheduled Script Instance Search, and get the list of the deployments that are ‘idle’

So, if deployments 1 and 2 are ‘in use’, I should get back 8 in the result set.

This should be easy no?

There are so many status, etc. Any insight? Much appreciated sir!

[TheUsualSuspect]

If you don’t mind, can you detail why you are manually handling deployments for map reduce?

[darrenhillconsulting]

Simply, I have a RESTlet that is capturing large amounts of data, and I want to hand off the processing to an available script deployment.

[darrenhillconsulting]

Below is the code (typescript) I’m currently using (not working quite right).

Code:
export let queueMRTask = (config: QueueMRTaskConfig): void => {
let i = 1;
while (i <= config.PROCESS_QUEUE_COUNT) {
try {
let msgProcessor: task.MapReduceScriptTask = task.create({
taskType: task.TaskType.MAP_REDUCE
});
msgProcessor.scriptId = config.scriptId;
msgProcessor.deploymentId = config.deploymentId_prefix + i.toString();
msgProcessor.params = config.parms;
let cassandraMsgProcessorTaskID = msgProcessor.submit();
let cassandraMsgProcessorTaskStatus: task.MapReduceScriptTaskStatus = task.checkStatus({
taskId: cassandraMsgProcessorTaskID
});

if (cassandraMsgProcessorTaskStatus.status !== task.TaskStatus.FAILED) {
i = config.PROCESS_QUEUE_COUNT + 1; // Force out of while look
} else {
i = i + 1;
}
} catch (e) {
i = i + 1;
}
}
};
As an example, I pass the following object into this function

Code:
config = {
PROCESS_QUEUE_COUNT: 10,
scriptId: ‘customscript_processor_mr’,
deploymentId_prefix: ‘customdeploy_processor_’,
parms: [1,2,3]
}
The deployment id’s following the naming conventions of

customdeploy_processor_1

customdeploy_processor_2

customdeploy_processor_3

.

.

.

[MChammaTX]

For me the Status for in Use are “Pending”, “Processing,”Retry”.

Also not sure if you’ve had a chance to check out SuiteCloud Processors, which actually replaces the whole queue model all together and may be a viable alternate solution.

---

darrenhillconsulting replied on 10/02/2017, 11:25 AM: I’m game for that, but see no documentation on how to trigger that via script. Anyone know where I can find it?

[david.smith]

Hey Darren. You can search for the current tasks and their progress.

Code:
var scheduledscriptinstanceSearchObj = search.create({
type: “scheduledscriptinstance”,
filters: [
[“startdate”,”on”,”today”],
“AND”,
[“mapreducestage”,”noneof”,”@NONE@”]
],
columns: [
search.createColumn({
name: “name”,
join: “script”
}),
search.createColumn({
name: “scriptid”,
join: “scriptDeployment”
}),
“timestampcreated”,
“mapreducestage”,
“status”,
search.createColumn({
name: “percentcomplete”,
sort: search.Sort.ASC
}),
search.createColumn({
name: “startdate”,
sort: search.Sort.DESC
}),
“enddate”,
“taskid”,
“queue”
]
});
var searchResultCount = scheduledscriptinstanceSearchObj.runPaged().count;
scheduledscriptinstanceSearchObj.run().each(function(result){
// .run().each has a limit of 4,000 results
return true;
});

[darrenhillconsulting]

I also read that, in SS1.0, if you omitted the deploymentId from nlapiScheduleScript (or passed a null), that Netsuite would use the next ‘available’ deployment. That would be ideal! Anyone know if the same holds true in SS2.0? I’ll give’r a go and let you know.

I appreciate all the help! Keep it coming!

[darrenhillconsulting]

Great news everyone, passing a ‘null’ for the deploymentid forces Netsuite to choose the next available deployment. Wicked!

[Author]

Posts