Changes between Version 10 and Version 11 of RemoteJob


Ignore:
Timestamp:
Jan 11, 2011, 3:45:28 AM (14 years ago)
Author:
tonig
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • RemoteJob

    v10 v11  
    11= Remote job submission =
     2
     3== Introduction and disclaimer ==
    24
    35A group from Universitat Pompeu Fabra
     
    1416to indicate the user of the RBoinc client and their machine.
    1517
    16 The system (Perl-based) is in boinc/rboinc/.
    1718
    1819'''Warning: this system has been used only by its developers.
    1920It will take some work to get it working on other projects.'''
    2021
    21 Powerpoint slides describing the system are
     22 * Powerpoint slides describing the system are
    2223[http://boinc.berkeley.edu/rboinc.pdf here].
    23 For details please see the paper T. Giorgino, M. J. Harvey and G. De Fabritiis, ''Distributed computing as a virtual supercomputer: Tools to run and manage large-scale BOINC simulations'', Comp. Phys. Commun. 181, 1402 (2010).  [[http://boinc.berkeley.edu/rboinc.pdf pdf]]
     24 * For details please see the paper T. Giorgino, M. J. Harvey and G. De Fabritiis, ''Distributed computing as a virtual supercomputer: Tools to run and manage large-scale BOINC simulations'', Comp. Phys. Commun. 181, 1402 (2010).  [[http://boinc.berkeley.edu/rboinc.pdf pdf]]
     25 * For client instructions see http://www.multiscalelab.org/utilities/RemoteBoinc
    2426
    2527== Summary ==
    2628
    27 (This section to be fixed.)
    28 The software should be fairly self-explanatory, but installation may be tricky. Here's a general overview
     29RBoinc is composed by the following main components
     30    1. Client scripts, which are used by the scientists to submit and retrieve jobs. They are boinc_retrieve and boinc_submit.
     31    1. Server cgis, used to handle files and interfacing with the rest of Boinc. They are called boinc_retrieve_server, boinc_submit_server
     32    1. Various (optional) monitoring scripts, which generate nightly reports, statistics, and the like.
    2933
    30     * boinc_retrieve_server, boinc_submit_server run as cgi. The former, actually, also handles all administrative requests (stop, purge).
    31     * boinc_retrieve, boinc_submit, are the client components (ditto as above for admin requests)
    32     * Exchange of files between client and server is done through WEBDAV http extensions (a scratch area needs be setup for this)
     34The software should be fairly self-explanatory, but installation may be tricky. The system is in boinc/rboinc/. Here's a general overview
    3335
     36    * You will need an apache web server on the Boinc server (either the existing one, or a separate process). This instance will serve
     37          * the RBOINC cgi-scripts, e.g. at http://YOURSERVER:8383/rboinc_cgi
     38          * a scratch area for temporary file exchage, exposed via WEBDAV, e.g. at  http://YOURSERVER:8383/DAV
    3439    * Wus naming is important and enforced like this: NNN-UUU_GGG-XX-YY-RNDzzzz where
    3540          * NN is the name of the workunit (sub-group)
     
    3944          * YY is the total n. of steps
    4045          * zzzz is a random number (not needed,actually)
    41     * WUs are kept in a "workflow_directory",  a subdir of the project dir,  as per slide 22 of the Powerpoint.
    42     * Inside each dir a "process" bash file is created, which is executed by the assimilator with the name of the assimilated WU as its argument. It will create_work the next step for execution.
    43     * The main reason for using perl is that  I preferred to use the XML::Simple module for (un-) xml-ing data structures over the network - it was useful for adding features on the fly keeping backwards compatibility
    44     * I implemented basic functions for authentication, but this is not finished yet
    45     * file storage is optimized through hardlinking and pooling. Network transfers are not (but they could be)
     46    * WUs are kept in a "workflow_directory",  a subdir of the project dir,  as per slide 22 of the Powerpoint. Inside each dir a "process" bash file is created, which is executed by the assimilator with the name of the assimilated WU as its argument. It will create_work the next step for execution.
     47    * File storage is optimized through hardlinking and pooling. (Network transfers are not yet)
     48    * Warning: authentication is not done yet (do secure the RBoinc port by firewall rules)
    4649
    4750
    48 == Annotating the WU template files ==
     51Both client and server are composed of Perl scripts (respectively command-line and cgi-bin). The main reason for using the Perl language is that I liked the XML::Simple module for (un-) xml-ing data structures over the network - which helped rapid development.
     52
     53== Client-side instructions ==
     54
     55
     56Instructions on using the client scripts are temporarily hosted at http://www.multiscalelab.org/utilities/RemoteBoinc .
     57
     58Client Perl scripts need be unpacked to some client-visible installation directory. Make sure your Perl installation fulfulls the dependencies (use ''cpan'' or your distribution's package manager if not).
     59
     60For details on the chaining mechanism, please see the paper T. Giorgino, M. J. Harvey and G. De Fabritiis, ''Distributed computing as a virtual supercomputer: Tools to run and manage large-scale BOINC simulations'', Comp. Phys. Commun. 181, 1402 (2010).  [[http://boinc.berkeley.edu/rboinc.pdf pdf]].
     61
     62
     63
     64== Server-side instructions ==
     65
     66The main steps to install the RBoinc server components are:
     67
     68 * Setup or adapt an  instance of the apache web server on the boinc server (or change the boinc one) to serve the rboinc cgi and DAV paths. See the ''apache.conf'' example file provided with the distribution. We shall assume that apache will serve at http://YOUR_SERVER:8383/rboinc_cgi
     69 * Copy the rboinc ''server'' scripts in the cgi directory, and edit the configuration file to suit your site setup.
     70 * You may want to revise the ''process'' script. It is invoked every time a WU is complete, to perform the submission of the next chain step.
     71 * Customize the WU and result template files, as directed below. This will RBoinc-enable Boinc ''applications'' of your choice.
     72 * If desired, install the SQL stored procedures (monitoring components).
     73
     74
     75=== Annotating the WU template files ===
    4976
    5077First, workunit template files should be marked as RBoinc-enabled at the top.
     
    111138
    112139
    113 == Annotating the result template files ==
     140=== Annotating the result template files ===
    114141
    115142Results template files are annotated with RBoinc-specific tags
     
    151178retrieve directory.
    152179
    153 
    154 
    155 For details on the chaining mechanism, please see the paper T. Giorgino, M. J. Harvey and G. De Fabritiis, ''Distributed computing as a virtual supercomputer: Tools to run and manage large-scale BOINC simulations'', Comp. Phys. Commun. 181, 1402 (2010).  [[http://boinc.berkeley.edu/rboinc.pdf pdf]]