When a microcompute transaction is executed using the DSUMO Computing API, the function(s) and dataset(s) involved in that transaction are passed to the web browsers of internet users for processing. The DSUMO Engine and DSUMO Framework are both loaded by those web browsers.
The DSUMO Engine handles communication with the DSUMO platform to establish timing metrics, track usage data, retrieve microcomute operations (the atoms of a microcompute transaction), and upload the results of those microcompute operations.
The DSUMO Framework is loaded by the DSUMO Engine and serves three purposes:
The functions described in this document are essential when using DSUMO Computing. They provide mechanisms for the parallelization of tasks, loop constructs and execution control commands.
Note that all the functions below are used in the form dsumo.[function]. The dsumo "namespace" is used to prevent conflict with any functions of the same name. dsumo is considered a reserved word, and should not be used in a DSUMO function except for the purpose of calling the functions described in this document.
In this document, three kinds of functions are discussed: referenced local functions, inline anonymous functions, and microcompute functions.
Referenced local functions are JavaScript functions that have already been defined. For example, this would include the functions declared below, both of which can be referenced as f:
Inline anonymous functions are JavaScript functions that are passed to DSUMO Framework calls as parameters. For example, this would be included in the use of dsumo.dwhile shown below:
Microcompute functions are those functions that are specific to DSUMO and which are defined within the DSUMO management portion of the KisoLabs website, or via the DSUMO Computing API. In all cases, microcompute functions are referenced by use of their name as a JavaScript string value. Microcompute function names are specific to each DSUMO service.
For example, a microcompute function named mf is referenced when calling dsumo.dreturn below:
All microcompute operations are executed within the scope of a container object which provides for sandboxing. This is transparent to the user of DSUMO Computing, and no special action need be taken on the part of the developer to leverage this container.
Amongst other functionality, the sandbox container overrides certain native JavaScript objects and functions that could result in the alteration of visual elements within a web page, or otherwise cause the end-user's browsing experince to be ill effected. Some of the objects and functions that are overridden are as follows:
window, document, alert, eval, XMLHttpRequest, XDomainRequest, Ajax, ActiveXObject, SWFObject
When called, these overridden objects will return a null value. If any sub-objects are referenced (e.g. document.location.href), an error will be thrown and execution of the microcompute operation will cease (which will likely cause the microcompute transaction to fail).
Example
Important: It is recognized that sandboxing within JavaScript is an unreliable model and may not offer full protection to the end user. The DSUMO Computing Terms of Use specifically prohibit any attempts to circumvent the protections that the sandbox affords. DSUMO Computing is intended for just that: Computing. Any use of DSUMO Computing that violates the Terms of Use may result in the termination of the involved KisoLabs/DSUMO account(s) and/or may result in legal action by KisoLabs, third party affiliates or government agencies. Please read and understand the Terms of Use fully before using DSUMO Computing.
The data container holds all "global scope" variables in DSUMO. Defining variables at the natively global (window) level is not possible for two reasons:
The data container is accessible using the keyword dc, which is a JavaScript object to which any sub-objects or values can be assigned.
Examples
The following lines of code demonstrate some valid uses of the data container. It is assumed that no previous declarations or assignments to these variables have been made.
After all of the above assignments have been made to the dc object those values will be available by referencing the dc object throughout the lifespan of a microcompute operation. However, be aware that the dc object does not persist after a call which executes a new microcompute operation (see Flow Control, below). However, the dc object may be explicitly passed to another microcompute operation.
Whereas the data container object can be thought of as a "global scope" within the context of a microcompute operation, the data set (ds) object is the equivalent of the argument parameter of the funtion called in a microcompute operation.
In other words, when using a flow control command which calls another DSUMO function (e.g. dsumo.dreturn([functionName]), dsumo.dparallel, and dsumo.dfork), the data passed to that flow control function will be available in the called function as the object with name ds.
It is vital to write code for DSUMO without assuming that variables passed between microcompute nodes as numeric types will be received as numeric types. This is not a limitation of DSUMO, but rather a result of the lowest-common capabilities between JSON parsers/stringifiers in different browsers and/or JavaScript frameworks.
For example, assume that you have returned the following using dsumo.dreturn with a chained call:
Below is the code for the function theNextFunction, which will receive the object passed to dsumo.dreturn as the object ds (the data set object):
Note that values passed into dsumo.dfor will be implicitly treated as numeric values for startValue, breakValue, and incrementor. Hence no "cast" function (such as parseInt) is necessary when passing integers-as-strings.
A key aspect to the adoption of DSUMO for Sites is that the DSUMO Framework can throttle the execution of JavaScript code to ensure that the processing of a microcompute operation takes no more than 10-15% of available CPU capacity. This prevents DSUMO from interfering with the ordinary workload of the internet user. The power of the DSUMO platform comes from its ability to put millions of instances of this fractional computing power to a single task -- your task.
JavaScript has no native ability to throttle code execution, and moreover has no native "sleep" capability. Even if it did, instances of a JavaScript engine usually run as a single threaded model and so any kind of "sleep" would block other functions from running. This includes all JavaScript code within the user's browser or browser tab. The same problem would occur with any long-running microcompute operation.
To overcome the lack of internal parallelism in JavaScript, a microcompute operation must periodically yield its place in JavaScript's execution stack. This is accomplished by implementing alternative loop functions other than the in-built for and while constructs for long-running loops.
Note: The use of native for and while loops are permitted within DSUMO functions, however they should not be used to encapsulate any code that will run for longer than a fraction of a millisecond. The DSUMO engine monitors the periods of time for which a microcompute operation runs without yielding, and if that time persistently runs long, the DSUMO function which was executing will be temporarily (and globally) invalidated.
Usage
The functional equivalent of the same operation using a native JavaScript for loop is shown below for comparison:
Parameters
| Parameter | Type | |
|---|---|---|
| counter | string | The name of the variable within the dc object that will be incremented (or decremented) by the loop on each iteration. |
| startValue | numeric | The counter will be set to this value during the first iteration of the loop. |
| breakValue | numeric | When the counter reaches this value, the loop will no longer be iterated (equivalent to using < with a positive incrementor or > with a negative incrementor in a typical for loop). |
| incrementor | numeric | This number will be added to the counter after each iteration of the loop. The value may be negative to decrement the counter. |
| function | function | The function that will be executed during each iteration of the loop. counter will be passed to the function
as its only parameter. function may be a reference to a local function, or may an anonymous function. For example, the parameter can take this form: function( [counter] ) { [loopBody] } counter - The value of the counter. (Optional) loopBody - The body of the function. |
| fNext | function (optional) | The function that will be executed upon completion of the loop. This may be a reference to a local function, or may be an anonymous function. |
Example 1 - Basic For Loop
This is a simple loop which will create a comma-delimited string containing each value of the counter. It demonstrates the use of inline anonymous functions and use of the counter as an argument to the function.
The above function will return the following string value:
Line 7: dsumo.dreturn is covered later in this document. Suffice to say it is somewhat equivalent to a standard JavaScript return, and it takes the value to be returned as its argument.
Example 2 - Basic For Loop with Local Functions
As in Example 1, this is a simple loop that will create a comma-delimited list of the value of the counter. However, it illustrates the use of functions that are referenced locally and uses the counter variable that is set to the dc object.
The above function will return the following string value:
Examples 1 and 2 accomplish the same goal, but have a difference in readability. Neither technique will yield a significant difference in performance, and it is up to the programmer to determine which is more appropriate for any particular task.
Example 3 - Nested For Loops
Similarly to the previous examples, these nested loops will create three comma-delimited lists showing the values of the counters.
The above function will return the following object:
The loop with counter i iterates three times, and so list_i contains the three values of the counter.
The loop with counter j iterates three times for every iteration of the loop with counter i, and so it iterates nine times in total. list_j therefore contains the three values of the counter j repeated three times.
list_ij demonstrates the interpolation of the values of i and j, and therefore has 12 elements.
dsumo.dwhile() operates differently from the standard JavaScript while loop as it does not take a conditional parameter for the conclusion of the loop. It will run indefinitely until dsumo.dbreak() is called.
Usage (dsumo.dwhile)
Parameters
| Parameter | Type | |
|---|---|---|
| function | function | The function that will be executed during each iteration of the loop. function may be a reference to a local function, or may an anonymous function. For example, the parameter can take this form: function() { [loopBody] } loopBody - The body of the function. |
| fNext | function (optional) | The function that will be executed upon completion of the loop. This may be a reference to a local function, or may be an anonymous function. |
Usage (dsumo.dbreak)
Parameters
This function does not accept any parameters. It must be called within the body of the function parameter of dsumo.dwhile() in order to cease iteration of the loop.
Note that dsumo.dbreak() can also be used within a dsumo.dfor loop to cease iteration of the loop.
Example 1 - While Loop
This is a simple loop that will create a comma-delimited list of integer values. It demonstrates the use of inline functions.
The above function will return the following string value:
There is no point to using DSUMO Computing without the flow control functions listed in this section. They provide for the parallel execution of microcompute operations, passing microcompute operation data between hosts, and returning results for later retrieval (or callback) from the computing API.
Usage
The primary purpose of dsumo.dreturn is to return the result of a microcompute function to a DSUMO service node. This result may then be collected via the DSUMO API (or in the case of a parallel function the result will be returned to the function specified when calling dsumo.dparallelStart).
Another use of dsumo.dreturn lies in the ability to specify another microcompute function to be called with the returned data as its data set object. This allows microcompute operations to be chained or to be called recursively across microcompute nodes.
Parameters
| Parameter | Type | |
|---|---|---|
| returnedData | (see description) | The data to be returned. The following Javascript types are acceptable: string, integer, float, Array, Object Important: When returning arrays or objects they must be serializable to JSON, even in the lowest-class implementation of a JSON stringifier. |
| function | string / function | If this parameter is passed as the JavaScript type of Function, it must refer to a local or anonymous function. In that
case two things will happen:
|
Return
This function has no local return value.
dsumo.dparallelStart, dsumo.dparallel, and dsumo.dfork
The few functions discussed in this section are the key to leveraging the power of the DSUMO platform. These functions enable your code to run on any arbitrary number of microcompute nodes simultaneously, and to pass data to and from those nodes.
Usage
Parameters
| Parameter | Type | |
|---|---|---|
| functionName | string | The function to which all functions called by dsumo.dparallel will return their data. The function name is passed as a string because it cannot be a local function, and must be a microcompute function. |
| data | object (optional) | A javascript object to be passed to each function called by dsumo.dparallel.
It is shared by all parallel microcompute operations, wheras data passed when calling
dsumo.dparallel is specific to each parallel microcompute operation. Important: If an index of the data object that is used when calling dsumo.dparallelStart is also used in the data object when calling dsumo.dparallel, it will be overridden by the value used when calling dsumo.dparallel. No error will be thrown. |
Return
| Name | Type | |
|---|---|---|
| Parallel Group ID | integer | Parallelized function calls are assigned to a group such that their results can be aggregated. |
Usage
Parameters
| Parameter | Type | |
|---|---|---|
| functionName | string | The function to be called for parallel execution. The function name is passed as a string because it cannot be a local function, and must be a microcompute function. |
| data | object | The JavaScript object that will be passed to the function specified by functionName. |
| parallelGroup | integer | The parallel group returned by the call to dsumo.dparallelStart. |
Return
This function has no return value.
Example 1
This is a simplified use of DSUMO's parallel capabilities. Implementing parallelism requires the referencing of at least three microcompute functions: The function which creates the parallel instances ("parallelCreator" in the example below), the function of which multiple instances will execute in parallel ("parallel"), and the function which will receive the results of each instance of the parallel function ("parallelReceptor").
Upon executing parallelCreator, it will create three instances of parallel, and immediately return the following string value:
After each instance of parallel have completed execution, parallelReceptor will be called. The variable ds will be available in parallelReceptor as an array. Each element of the array will be the return value of each instance of parallel.
parallelReceptor will return the following string value:
Important: Parallel microcompute operations are not executed until dsumo.dreturn is called within the function which creates the parallel instances.
Example 2 - Bad Practice
The example below is given to demonstrate a particularly bad practice when using the DSUMO Framework's parallel execution control functions: Passing common data to each parallel instance individually, instead of to dsumo.dparallelStart. The code is mostly identical to that of Example 1, with the exception of the text shown in red. It can be assumed that the functions "parallel" and "parallelReceptor" are the same as in Example 1.
In any case where data will be used across all (or most) parallel microcompute operations, it is significantly more efficient to pass that data to those parallel operations using dsumo.dparallelStart, rather than in each call to dsumo.dparallel.
Unlike dsumo.dparallelStart / dsumo.dparallel, the results from dsumo.dfork are not aggregated and returned to a single function. The results of each forked instance are stored independantly for retrieval (or callback) from the DSUMO API.
Usage
Parameters
| Parameter | Type | |
|---|---|---|
| data | object | A JavaScript object to be passed to the function specified. |
| functionName | string | The microcompute function to be called for execution. The function name is passed as a string because it cannot be a local function. |
Return
This function has no return value.
Example 1
This example creates three forks, two of which are processed by fork0 and one of which is processed by fork1. Both of these functions do a simple string concatenation before returning their results.
The function forkCreator will execute fork0 twice, then fork1 once, and immediately return the string value:
After each instance of the forked functions have completed execution, they will return their result for later retrieval (or callback) via the API.
The results of the forked functions are returned as strings:
As in many programming languages, the DSUMO Framework offers the concept of "includes". Unlike in most languages, there are two distinct types of includes: Microcompute function includes and library includes.
A Microcompute Function may include another microcompute function (and these references may occur to any arbitrary depth).
These includes are treated as code-inline, similarly to a PHP include directive. The DSUMO include directive may appear anywhere within a function.
Usage
//dinclude:[Function Name]
The include directive begins with the comment characters // such that it is not parsed by the DSUMO JavaScript validator.
Note: If the Function Name referenced does not exist, a fatal error will be thrown by the microcompute operation under which the function executes. This will likely cause the encapsulating microcompute transaction to fail.
Example
This example demonstrates the inline nature of the dinclude directive. It is assumed that the function iAmIncluded has been defined under the same DSUMO Computing service that has executed the microcompute function named aMicrocomputeFunction.
Before the function aMicrocomputeFunction is executed, the contents of iAmIncluded is inserted textually. As such the following function will be interpreted by the microcompute node:
After execution of aMicrcomputeFunction has completed, the following JSON string is returned:
Note that the variables i and s were declared in the include, but were treated as if they were declared in aMicrocomputeFunction. Likewise with the function includeTest.
Unlike microcompute function includes, library includes are not treated as code-inline. These includes are analagous to <SCRIPT src=""> tags in HTML, and in fact do cause a microcompute node to fetch a file containing JavaScript code.
To prevent arbitrary execution of JavaScript code from external servers, library includes must be retrieved from DSUMO's servers. A catalog of available libraries is available here.
Usage
//linclude:[Relative Path]
The linclude directive may be located anywhere within a microcompute function, however any libraries referenced will be loaded by the microcompute node before the microcompute function is executed. If an linclude directive appears more than once referencing the same library, that library will only be loaded once. As such it is of no concern if, for example, the same library is referenced in two functions that are dincluded in the same microcompute operation.
Once a library has been included, it is then addressable within a microcompute function using the prefix dlib.
Library Usage Pattern
dlib.[Library Name]
Above is a generalized pattern for the usage of functions included in a library. See the example below for further elucidation.
Important: In the example below, the object Example is defined within the scope of the object DSUMOLib. Within the microcompute function it is then referenced as being within the object dlib. This is not an error in the documentation. Microcompute operations are processed within the scope of a container object; The library object (DSUMOLib) is passed to the container as dlib.
Example
The following is a Microcompute Function executed in a Microcompute Operation. For the purposes of this example, it is assumed that the library /misc/test/Example.js is available from DSUMO.
The content of /misc/test/Example.js is assumed to be as follows:
This string value will be returned by the microcompute operation:
Libraries available to microcompute functions will leverage the DSUMO Framework, and therefore may have special implementation requirements. Please review the documentation for each library carefully before you use it!