Context Navigation

Changes between Version 56 and Version 57 of RemoteJobs

Timestamp:: Feb 24, 2017, 2:20:30 PM (7 years ago)
Author:: davea
Comment:: --

Legend:

: Unmodified
: Added
: Removed
: Modified

RemoteJobs

-                      v56
+                      v57
 [[PageOutline]]
 = RPCs for remote job submission =
+This document describes APIs for remotely submitting,
+monitoring, and controlling jobs on a BOINC server.
+The APIs supports the submission of '''batches''' of jobs,
+which may contain a single job or many thousands of jobs.
+Currently, the API has two restrictions:
+This document describes APIs for remotely submitting, monitoring, and controlling jobs on a BOINC server. The APIs supports the submission of '''batches''' of jobs, which may contain a single job or many thousands of jobs. Currently, the API has two restrictions:
  * All jobs in a batch must use the same application.
  * There can be no dependencies between jobs.
+At the bottom level, the interface consists of Web RPCs.
+BOINC provides client-side bindings in PHP, C++, and Python.
+These bindings differ slightly; they expose different details of the Web RPCs.
+Interfaces for [RemoteInputFiles staging input files]
+and [RemoteOutputFiles fetching output files] are described separately.
+There are various options for managing input files.
+If you use [RemoteInputFiles#Job-basedfilemanagement Job-based file management],
+which maintains batch/file associations,
+the order of operations is:
+At the bottom level, the interface consists of Web RPCs. BOINC provides client-side bindings in PHP, C++, and Python. These bindings differ slightly; they expose different details of the Web RPCs.
+Interfaces for [wiki:RemoteInputFiles staging input files] and [wiki:RemoteOutputFiles fetching output files] are described separately.
+There are various options for managing input files. If you use [wiki:RemoteInputFiles#Job-basedfilemanagement Job-based file management], which maintains batch/file associations, the order of operations is:
  * Create a batch (initially empty); returns the batch ID.
 …
  * Submit jobs, passing the batch ID
+If you manage input files a different way,
+then you create the batch and submit jobs in a single API call.
+If you manage input files a different way, then you create the batch and submit jobs in a single API call.
 Once you have submitted the batch, you can
  * '''Monitor''' the batch with query_batches(), query_batch(), or query_job().
+ * '''Abort''' the batch (if you see errors, or if enough jobs have been finished)
+  using abort_jobs() or abort_batch().
+ * [RemoteOutputFiles download output files].
+ * '''Retire''' the batch using retire_batch().
+  This tells the server to clean up the files and job records associated with the batch,
+  and to mark the batch as "retired";
+  retired batches are normally not shown in the web interface.
+ * '''Abort''' the batch (if you see errors, or if enough jobs have been finished) using abort_jobs() or abort_batch().
+ * [wiki:RemoteOutputFiles download output files].
+ * '''Retire''' the batch using retire_batch(). This tells the server to clean up the files and job records associated with the batch, and to mark the batch as "retired"; retired batches are normally not shown in the web interface.
 == PHP interface ==
+The following functions are provided in the PHP file
+[/trac/browser/trunk/boinc/html/inc/submit.inc submit.inc],
+which is independent of other BOINC PHP code.
+The file html/user/submit_test.php has code to exercise and test these functions.
+The following functions are provided in the PHP file [/trac/browser/trunk/boinc/html/inc/submit.inc submit.inc], which is independent of other BOINC PHP code. The file html/user/submit_test.php has code to exercise and test these functions.
 === boinc_submit_batch() ===
 Submits a batch.
 Arguments: a "request object" whose fields include
  * '''project''': the project URL
  * '''authenticator''': the user's authenticator
  * '''app_name''': the name of the application for which jobs are being submitted
+ * '''batch_name''': a symbolic name for the batch.
+   Must be unique.
+   If omitted, a name of the form "batch_unixtime" will be used.
+ * '''workunit_template_file''': an optional [JobTemplates input template file name].
+ * '''result_template_file''': an optional [JobTemplates output template file name].
+ * '''batch_name''': a symbolic name for the batch. Must be unique. If omitted, a name of the form "batch_unixtime" will be used.
+ * '''workunit_template_file''': an optional [wiki:JobTemplates input template file name].
+ * '''result_template_file''': an optional [wiki:JobTemplates output template file name].
  * '''jobs''': an array of job descriptors, each of which contains
+  * '''rsc_fpops_est''': an estimate of the FLOPs used by the job
+  * '''command_line''': command-line arguments to the application
+  * '''wu_template''': optional; the [http://boinc.berkeley.edu/trac/wiki/JobTemplates#Inputtemplates input template]
+    to use for this job, as an XML string.
+  * '''result_template''': optional;
+    the [http://boinc.berkeley.edu/trac/wiki/JobTemplates#Outputtemplates output template]
+    to use for this job, as an XML string.
+  * '''input_files''': an array of input file descriptors, each of which contains
+   * '''mode''': "local", "semilocal", "local_staged", "inline", or "remote" (see below).
+   * '''source''': meaning depends on mode:
+    * local: path on the BOINC server
+    * semilocal: the file's URL
+    * local_staged: physical name
+    * inline: the file's contents
+   * For "remote" mode, instead of "source" you must specify:
+    * '''url''': the file's URL
+    * '''nbytes''': file size
+    * '''md5''': the file's MD5 checksum
+   * '''rsc_fpops_est''': an estimate of the FLOPs used by the job
+   * '''command_line''': command-line arguments to the application
+   * '''wu_template''': optional; the [http://boinc.berkeley.edu/trac/wiki/JobTemplates#Inputtemplates input template] to use for this job, as an XML string.
+   * '''result_template''': optional; the [http://boinc.berkeley.edu/trac/wiki/JobTemplates#Outputtemplates output template] to use for this job, as an XML string.
+   * '''input_files''': an array of input file descriptors, each of which contains
+     * '''mode''': "local", "semilocal", "local_staged", "inline", or "remote" (see below).
+     * '''source''': meaning depends on mode:
+       * local: path on the BOINC server
+       * semilocal: the file's URL
+       * local_staged: physical name
+       * inline: the file's contents
+     * For "remote" mode, instead of "source" you must specify:
+       * '''url''': the file's URL
+       * '''nbytes''': file size
+       * '''md5''': the file's MD5 checksum
 Result: a 2-element array containing
  * The batch ID
  * An error message (null if success)
 Input files can be supplied in any of the following ways:
+ * '''local''': the file is on the BOINC server and is not [JobStage staged].
+  It's specified by its full path.
+ * '''local_staged''': the filed has been [JobStage staged] on the BOINC server.
+  It's specified by its physical name.
+ * '''semilocal''': the file is on a data server that's accessible to the BOINC server
+   but not necessarily to the outside world.
+   The file is specified by its URL.
+   It will be downloaded by the BOINC server during job submission,
+   and served to clients from the BOINC server.
+ * '''inline''': the file is included in the job submission request XML message.
+   It will be served to clients from BOINC server.
+ * '''remote''': the file is on a data server other than the BOINC server,
+   and will be served to clients from that data server.
+   It's specified by the URL, the file size, and the file MD5.
+ * '''local''': the file is on the BOINC server and is not [wiki:JobStage staged]. It's specified by its full path.
+ * '''local_staged''': the filed has been [wiki:JobStage staged] on the BOINC server. It's specified by its physical name.
+ * '''semilocal''': the file is on a data server that's accessible to the BOINC server but not necessarily to the outside world. The file is specified by its URL. It will be downloaded by the BOINC server during job submission, and served to clients from the BOINC server.
+ * '''inline''': the file is included in the job submission request XML message. It will be served to clients from BOINC server.
+ * '''remote''': the file is on a data server other than the BOINC server, and will be served to clients from that data server. It's specified by the URL, the file size, and the file MD5.
 The following mode has been proposed but is not implemented yet:
+ * '''sandbox''': the file is in the user's [RemoteInputFiles#Per-userfilesandbox sandbox],
    and is specified by its name in the sandbox.
+ * '''sandbox''': the file is in the user's [wiki:RemoteInputFiles#Per-userfilesandbox sandbox], and is specified by its name in the sandbox.
 The following example submits a 10-job batch:
 {{{
 $req = new StdClass;
 …
+}
 }}}
+Note: this interfaces is inconsistent; it lets you do some things but not others.
+Let me know if you need additions.
+Note: this interfaces is inconsistent; it lets you do some things but not others. Let me know if you need additions.
 === boinc_estimate_batch() ===
 Returns an estimate of the elapsed time required to complete a batch.
+Arguments: same as '''boinc_submit_batch()'''
+(only relevant fields need to be populated).
+Arguments: same as '''boinc_submit_batch()''' (only relevant fields need to be populated).
 Return value: a 2-element array containing
  * The elapsed time estimate, in seconds
  * An error message (null if success)
 === boinc_query_batches() ===
 Returns a list of this user's batches, both in progress and complete.
 Argument: a request object with elements
  * '''project''' and '''authenticator''': as above.
  * '''get_cpu_time''' (optional): if nonzero, get CPU time of each batch
+Result: a 2-element array.
+The first element is an array of batch descriptor objects,
+each with the following fields:
+Result: a 2-element array. The first element is an array of batch descriptor objects, each with the following fields:
  * '''id''': batch ID
  * '''state''': values are
   * 1: in progress
   * 2: completed (all jobs either succeeded or had fatal errors)
   * 3: aborted
   * 4: retired
+   * 1: in progress
+   * 2: completed (all jobs either succeeded or had fatal errors)
+   * 3: aborted
+   * 4: retired
  * '''name''': the batch name
  * '''app_name''': the application name
 …
  * '''nerror_jobs''': the number of jobs that had fatal errors
  * '''completion_time''': when the batch was completed
+ * '''credit_estimate''': BOINC's initial estimate of the credit that
+  would be granted to complete the batch, including replication
+ * '''credit_estimate''': BOINC's initial estimate of the credit that would be granted to complete the batch, including replication
  * '''credit_canonical''': the actual credit granted to canonical instances
  * '''credit_total''': the actual credit granted to all instances
 === boinc_query_batch() ===
 Gets batch details.
 Argument: a request object with elements
  * '''project''' and '''authenticator''': as above
  * '''batch_id''': specifies a batch.
  * '''get_cpu_time''' (optional): if nonzero, get CPU time of batch
+ * '''get_cpu_time''' (optional): if nonzero, get CPU time of batch. This includes all job instances, and doesn't include GPU time, so it may not be meaningful.
  * '''get_job_details''' (optional): if nonzero, return job details (see below).
+Result: a 2-element array.
+The first element is a batch descriptor object as described above,
+with one additional field:
+Result: a 2-element array. The first element is a batch descriptor object as described above, with one additional field:
  * '''jobs''': an array of job descriptor objects, each one containing
   * '''id''': the database ID of the job's workunit record
   * '''canonical_instance_id''': if the job has a canonical result, its database ID
+   * '''id''': the database ID of the job's workunit record
+   * '''canonical_instance_id''': if the job has a canonical result, its database ID
 If '''get_job_details''' was set, the job descriptors also contain:
+  * '''status''': "unsent", "in_progress", "error", or "done".
+  * '''cpu_time''': if done, the CPU time
+  * '''exit_status''': if error, the exit status of one of the error instances.
+ * '''status''': "unsent", "in_progress", "error", or "done".
+ * '''cpu_time''': if done, the CPU time
+ * '''exit_status''': if error, the exit status of one of the error instances.
 The order of job descriptors matches their order in the batch submission.
 === boinc_query_job() ===
 Gets job details.
 Argument: a request object with elements:
  * '''project''' and '''authenticator''': as above
  * '''job_id''': specifies a job.
 Result: a 2-element array.
+The first element is a job descriptor object with the following fields:
+Result: a 2-element array. The first element is a job descriptor object with the following fields:
  * '''instances''': an array of job instance descriptors, each containing:
+  * '''name''': the instance's name
+  * '''id''': the ID of the corresponding result record
+  * '''state''': a string describing the instance's state
+   (unsent, in progress, complete, etc.)
+  * '''outfile''': if the instance is over,
+   a list of output file descriptors, each containing
+   * '''size''': file size in bytes
+   * '''name''': the instance's name
+   * '''id''': the ID of the corresponding result record
+   * '''state''': a string describing the instance's state (unsent, in progress, complete, etc.)
+   * '''outfile''': if the instance is over, a list of output file descriptors, each containing
+     * '''size''': file size in bytes
 === boinc_abort_batch() ===
 Argument: a request object with elements
  * '''project''' and '''authenticator''': as above,
  * '''batch_id''': specifies a batch.
 …
 === boinc_retire_batch() ===
 Delete server storage (files, DB records) associated with a batch.
 Argument: a request object with elements
  * '''project''' and '''authenticator''': as above,
  * '''batch_id''': specifies a batch.
 …
 == C++ interface ==
+A C++ interface to the following functions is available in lib/remote_submit.cpp.
+Include lib/remote_submit.h.
+All functions return zero on success,
+else an error code as defined in lib/error_numbers.h
+A C++ interface to the following functions is available in lib/remote_submit.cpp. Include lib/remote_submit.h.
+All functions return zero on success, else an error code as defined in lib/error_numbers.h
 === create_batch() ===
 Create a batch - a set of jobs, initially empty.
 {{{
 int create_batch(
 …
 );
 }}}
  project_url:: the project URL
  authenticator:: the authenticator of the submitting user
  batch_name:: a name for the batch.  Must be unique over all batches.
  app_name:: the name of an application on the BOINC server
+ expire_time:: if nonzero, the Unix time when the batch should
+   be aborted and removed from the server, whether or not it's completed.
+ expire_time:: if nonzero, the Unix time when the batch should be aborted and removed from the server, whether or not it's completed.
  batch_id:: (out) the batch's database ID
  error_msg:: (out) an error message if the operation failed
 === estimate_batch() ===
 Get an estimate of the makespan of a (potential) batch.
 …
 );
 }}}
  jobs :: description of jobs; same as for '''submit_jobs()''' (see below).
  est_makespan :: the estimated makespan (time to completion).
+ jobs:: description of jobs; same as for '''submit_jobs()''' (see below).
+ est_makespan:: the estimated makespan (time to completion).
 === submit_jobs() ===
 Submit a set of jobs; place them in an existing batch, and make them runnable.
 {{{
 int submit_jobs(
 …
 For each job:
  job_name:: must be unique over all jobs
  cmdline_args:: command-line arguments
 …
 For each input file:
  physical_name:: BOINC's physical name for the file.  The file must already be staged.
 === query_batches() ===
 Query the status of this user's batches.
 …
 };
 }}}
 === query_batch() ===
 Return the detailed status of jobs in a given batch (can specify by either ID or name).
 …
 }}}
 === abort_jobs() ===
 Abort a set of jobs.
 {{{
 extern int abort_jobs(
 …
 }}}
 === query_completed_job() ===
 Query a completed job.
 {{{
 extern int query_completed_job(
 …
 };
 }}}
  canonical_resultid:: database ID of the "canonical" instance of the job.
  error_mask:: a bitmask of error conditions (see db/boinc_db_types.h)
 …
 === retire_batch() ===
+"Retire" a batch.
+The server is then allowed to delete the batch's input and output files,
+and its database records.
+"Retire" a batch. The server is then allowed to delete the batch's input and output files, and its database records.
 {{{
 extern int retire_batch(
 …
 );
 }}}
 === set_expire_time() ===
 Change the expiration time of a batch.
 {{{
 extern int set_expire_time(
 …
 );
 }}}
 === ping_server() ===
 Ping the project's server; return zero if the server is up.
 {{{
 extern int ping_server(
 …
 );
 }}}
 === query_batch_set() ===
+Return the status of the jobs in a given set of batches.
+This is used by the Condor interface; it's probably not useful outside of that.
+Return the status of the jobs in a given set of batches. This is used by the Condor interface; it's probably not useful outside of that.
 {{{
 extern int query_batch_set(
 …
 }}}
 == Python binding ==
+The file tools/submit_api.py contains a Python binding of some of above RPCs.
+For examples of its use, see tools/submit_api_test.py.
+To build a description of a batch of jobs,
+use the BATCH_DESC, JOB_DESC, and FILE_DESC classes:
+The file tools/submit_api.py contains a Python binding of some of above RPCs. For examples of its use, see tools/submit_api_test.py.
+To build a description of a batch of jobs, use the BATCH_DESC, JOB_DESC, and FILE_DESC classes:
 {{{
 def make_batch():
 …
 }}}
 You can then pass this to either estimate_batch() or submit_batch():
 {{{
     from submit_api import *
 …
     print 'estimated time: ', r.text, ' seconds'
 }}}
+The return value of all the API functions is an !EntityTree representation
+of the XML returned by the RPC.
+Other requests use a REQUEST object.
+For example, to query the status of batch 271:
+The return value of all the API functions is an !EntityTree representation of the XML returned by the RPC.
+Other requests use a REQUEST object. For example, to query the status of batch 271:
 {{{
     req = REQUEST()
 …
 }}}
 Possible attributes of FILE_DESC:
  * mode
  * url
 …
 Possible attributes of JOB_DESC:
  * rsc_fpops
  * command_line
 …
 Possible attributes of BATCH_DESC:
  * project (URL of project)
  * authenticator (submitter's account key)
 …
  * jobs (list of JOB_DESC)
+Available functions are listed below.
+Each function takes a request object whose attributes include
+at least project and authenticator.
+ abort_batch(req)::
+    req attributes: batch_id
+ abort_jobs(req)::
+    req attributes: jobs (list of job names)
+ create_batch(req)::
+    req attributes: app_name, batch_name, expire_time
+ estimate_batch(req)::
+    req is a BATCH_DESC
+ query_batch(req)::
+    req attributes: batch_id, get_cpu_time
+ query_batches(req)::
+    req attributes_ get_cpu_time
+ query_completed_job(req)::
+    req attributes: job_name
+ query_job(req)::
+    req attributes: job_id
+ get_output_file(req)::
+    req attributes: instance_name, file_num
+ get_output_files(req)::
+    req attributes: batch_id
+ retire_batch(req)::
+    req attributes: batch_id
+Available functions are listed below. Each function takes a request object whose attributes include at least project and authenticator.
+ abort_batch(req):: req attributes: batch_id
+ abort_jobs(req):: req attributes: jobs (list of job names)
+ create_batch(req):: req attributes: app_name, batch_name, expire_time
+ estimate_batch(req):: req is a BATCH_DESC
+ query_batch(req):: req attributes: batch_id, get_cpu_time
+ query_batches(req):: req attributes_ get_cpu_time
+ query_completed_job(req):: req attributes: job_name
+ query_job(req):: req attributes: job_id
+ get_output_file(req):: req attributes: instance_name, file_num
+ get_output_files(req):: req attributes: batch_id
+ retire_batch(req):: req attributes: batch_id
 == HTTP/XML interface ==
+At a lower level, the APIs are accessed by sending a POST request,
+using HTTP or HTTPS, to PROJECT_URL/submit_rpc_handler.php.
+The inputs and outputs of each function are XML documents.
+The format of the request and reply XML documents
+can be inferred from user/submit_rpc_handler.php.
+At a lower level, the APIs are accessed by sending a POST request, using HTTP or HTTPS, to PROJECT_URL/submit_rpc_handler.php. The inputs and outputs of each function are XML documents. The format of the request and reply XML documents can be inferred from user/submit_rpc_handler.php.
 == Example web interface ==
+An example of a web interface for job submission and control,
+based on this API, can be found here:
+http://boinc.berkeley.edu/trac/browser/trunk/boinc/html/user/submit_example.php
+This example is functional and it shows how to use the API.
+However, you will have to modify it heavily for your particular
+applications and web site.
+An example of a web interface for job submission and control, based on this API, can be found here: http://boinc.berkeley.edu/trac/browser/trunk/boinc/html/user/submit_example.php
+This example is functional and it shows how to use the API. However, you will have to modify it heavily for your particular applications and web site.