Differences between revisions 1 and 2
Revision 1 as of 2012-12-31 13:42:12
Size: 6650
Comment:
Revision 2 as of 2013-01-01 19:34:51
Size: 8341
Comment: Overhaul
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
This page explains how to create units of work for one's self-run BOINC project.
Line 10: Line 12:
A work unit is the portion of data that the project should be analysed. It has the following parts, A work unit is a portion of the project's data that the project should be analysed with a particular application. Applications and data can be shared between work units. The following parts need to be known for the creation of a workunit in the database.
Line 17: Line 19:
Input files are what the application should be working on. But typically those vary for every work unit, like the range of numbers in which to search for primes. We will learn at some later stage that such inputs can be shared between work units, like a particular protein structure against which a large set of ligands shall be tested for their molecular interaction.

For this start, we manually create an invariant test input file. It is a one-lined arbitrary string, stored in $installroot/$fileprojectname/download/input_file in all lower case. The application run will convert it into all upper case.

{{{
cd $installroot/$fileprojectname && echo test string >> download/input_file || echo "Something went wrong, check variables."
Input files are what the application should be working on. Typically those vary for every work unit, like the range of numbers in which to search for primes. We will learn at some later stage that such inputs can be shared between work units, like a particular protein structure against which a large set of ligands shall be tested for their molecular interaction.

The example application this tutorial addresses converts files in mixed case to a file with all letters in upper case. For this, we manually create an invariant test input file. It is a one-lined arbitrary string, stored in $installroot/$fileprojectname/download/input_file in all lower case. The application run will convert it into all upper case.

{{{
. ~/.boinc_test.conf
if test -n "$installroot" -a -n "$fileprojectname" &&
cd $installroot/$fileprojectname; then
   sudo strace -o download/input_file -- strace -h > /dev/null
&& echo "[ok]" || echo "[failed]"
else
   echo "E: please
check parameters in ~/.boinc_test.conf"
fi
Line 30: Line 37:
cp /usr/share/doc/boinc-server-maker/examples/upper_case_* $installroot/$fileprojectname/templates/ sudo cp /usr/share/doc/boinc-server-maker/examples/upper_case_{result,wu} $installroot/$fileprojectname/templates/
Line 40: Line 47:
== Fill the database ==

We will now introduce the system to a single work unit - only a single. The input remains described in the file ''input_file'', templates are specified individually for result and work unit (wu). The application name is read from the command line, not from the wu description. This way, one could have different applications work on the same work unit more easily. Every unit is also passed a name, which the user sees in the BOINC manager and shall identify the work unit.

We will call the workunit test and to commit it to the database, execute
{{{
cd $installroot/$fileprojectname && \
./bin/create_work -appname upper_case -wu_name test -wu_template templates/upper_case_wu -result_template templates/upper_case_result input_file
}}}

== Uploading multiple inputs ==

''FIXME: this should be "multiple workunits", not "multiple inputs, right?"''

All input files need to be created at some stage. So they will be available, most likely as some file or they can be made available as a file. For batch invocations of ''create_work'', one can use the shell's loop functionality. For the BASH, this could for instance be {{{
for i in $(seq 1 3); do
  echo "./bin/create_work -appname upper_case -wu_name test$i input_${i}_file"
done
}}}

yielding{{{
./bin/create_work -appname upper_case -wu_name test1 input_1_file
./bin/create_work -appname upper_case -wu_name test2 input_2_file
./bin/create_work -appname upper_case -wu_name test3 input_3_file
}}}

== Inspections (optional) ==

=== Files created ===

The download folder will now have a directory created especially for this workunit. To find this (or other modifications after the execution of some BOINC tool) seek the latest modified folder: {{{
$ ls -ltr $installroot/$fileprojectname
drwxrwxr-x 2 root boincadm 4096 Jul 9 13:40 templates
drwxrwxr-x 3 root boincadm 4096 Jul 9 14:09 download
}}}
and in download this is {{{
$ ls -ltr $installroot/$fileprojectname/download | tail -1
drwxrwx--x 2 root root 4096 Jul 9 14:09 76
}}}
which again shows {{{
 ls -ltr $installroot/$fileprojectname/download/76
total 4
-rw-r--r-- 1 root root 12 Jul 9 14:09 input_file
}}}
which is the very same file.

=== Database inspection ===

{{{
mysql> select id,create_time,appid,name,fileset_id from workunit;
| id | create_time | appid | name | fileset_id |
+----+-------------+-------+------+------------+
| 1 | 1310213354 | 11 | test | 0 |
}}}

= Multifile Applications =

== Multiple Binaries ==

The applications which consists of more than one files, should be arranged in the following hierarchy for them to be parsed by ''bin/update_versions''
{{{
apps/
`-- appname
    |-- appname_version_platformname1
    | `-- FILES
    |-- appname_version_platformname2
        `-- FILES
}}}
Then do
{{{
$ bin/update_versions
}}}
to commit to the database.

Please see [[BOINC/WrapperApp]] deployment page, which is a multifile BOINC application.

== Multiple Input/Output Files ==

The input/output files are specified in the workunit and result templates respectively. The '''<file_info>''', '''<file_ref>''' tag pairs each describes a file. For example, if we need to create a workunit for ''test'' application which takes ''in1'', ''in2'' as input files, the workunit template would be

{{{
$ cat templates/test_wu
<file_info>
  <number>0</number>
</file_info>
<file_info>
  <number>1</number>
</file_info>

<workunit>
  <file_ref>
    <file_number>0</file_number>
    <open_name>in1</open_name>
  </file_ref>
  <file_ref>
    <file_number>1</file_number>
    <open_name>in2</open_name>
  </file_ref>
</workunit>
}}}

Please see each file has a '''<file_info>''' and '''<file_ref>''' tag. The same tags are used in case of result template,i.e if our application generates two output files, ''out1'', ''out2'' which should be transported back,
The result template would be
{{{
$ cat template/test_result
The files are not overly complicated. In their basic form, this means. There is a [[http://boinc.berkeley.edu/trac/wiki/JobSubmission|job sumission documentation]] on the BOINC Wiki indicating much more thoughts to possibly be put into it all: {{{
$ cat /var/tmp/boinc-server-test/boinctest/templates/upper_case_result
Line 153: Line 56:
<result>
  <file_ref>
    <file_name><OUTFILE_0/></file_name>
    <open_name>out</open_name>
  </file_ref>
</result>
}}}

and similarly {{{
$ cat /var/tmp/boinc-server-test/boinctest/templates/upper_case_wu
<file_info>
  <number>0</number>
</file_info>
<workunit>
  <file_ref>
    <file_number>0</file_number>
    <open_name>in</open_name>
  </file_ref>
  <min_quorum>1</min_quorum>
  <target_nresults>1</target_nresults>
</workunit>
}}}

which with some confidence is working for quite a number of different projects.

== Fill the database ==

We will now introduce the system to a single work unit - only a single. The input remains described in the file ''input_file'', templates are specified individually for result and work unit (wu). The application name is read from the command line, not from the wu description. This way, one could have different applications work on the same work unit more easily. Every unit is also passed a name, which the user sees in the BOINC manager and shall identify the work unit.

We will call the workunit test and to commit it to the database, execute
{{{
[ -n "$installroot" -a -n "$fileprojectname" ] && \
cd $installroot/$fileprojectname && \
sudo ./bin/create_work -appname upper_case -wu_name test -wu_template templates/upper_case_wu -result_template templates/upper_case_result input_file || echo "E: something failed"
}}}

== Uploading multiple work units ==

All input files need to be created at some stage. So they will be available, most likely as some file or they can be made available as a file. For batch invocations of ''create_work'', one can use the shell's loop functionality. For the BASH, this could for instance be {{{
for i in $(seq 1 3); do
  echo ./bin/create_work -appname upper_case -wu_name test$i input_${i}_file
done
}}}

yielding{{{
./bin/create_work -appname upper_case -wu_name test1 input_1_file
./bin/create_work -appname upper_case -wu_name test2 input_2_file
./bin/create_work -appname upper_case -wu_name test3 input_3_file
}}}

== Inspections (optional) ==

=== Files created ===

The download folder will now have a directory created especially for this workunit. To find this (or other modifications after the execution of some BOINC tool) seek the latest modified folder: {{{
$ ls -ltr $installroot/$fileprojectname
...
drwxrwxr-x 2 root boincadm 4096 Jul 9 13:40 templates
drwxrwxr-x 3 root boincadm 4096 Jul 9 14:09 download
}}}
and in download this is {{{
$ ls -ltr $installroot/$fileprojectname/download | tail -1
drwxrwx--x 2 root root 4096 Jul 9 14:09 76
}}}
which again shows {{{
 ls -ltr $installroot/$fileprojectname/download/76
total 4
-rw-r--r-- 1 root root 12 Jul 9 14:09 input_file
}}}
which is the very same file.

=== Database inspection ===

{{{
mysql> select id,create_time,appid,name,fileset_id from workunit;
| id | create_time | appid | name | fileset_id |
+----+-------------+-------+------+------------+
| 1 | 1310213354 | 11 | test | 0 |
}}}

= Multifile Applications =

== Multiple Binaries ==

With the renewed application-version-platform hierarchy outlined earlier, applications that consists of more than one file are already covered. We will get back to it in the context of legacy applications being wrapped for BOINC compatibility, which needs two apps - the wrapper and the legacy one. Please skim through [[BOINC/WrapperApp]] to mentally prepare for it.

== Multiple Input/Output Files ==

The input/output files are specified in the workunit and result templates respectively. The '''<file_info>''', '''<file_ref>''' tag pairs each describes a file. For example, if we need to create a workunit for ''test'' application which takes ''in1'', ''in2'' as input files, the workunit template would be

{{{
$ cat templates/test_wu
<file_info>
  <number>0</number>
</file_info>
<file_info>
  <number>1</number>
</file_info>

<workunit>
  <file_ref>
    <file_number>0</file_number>
    <open_name>in1</open_name>
  </file_ref>
  <file_ref>
    <file_number>1</file_number>
    <open_name>in2</open_name>
  </file_ref>
</workunit>
}}}

Please see each file has a '''<file_info>''' and '''<file_ref>''' tag. The same tags are used in case of result template,i.e if our application generates two output files, ''out1'', ''out2'' which should be transported back,
The result template would be
{{{
$ cat template/test_result

<file_info>
  <name><OUTFILE_0/></name>
  <generated_locally/>
  <upload_when_present/>
  <max_nbytes>10000</max_nbytes>
  <url><UPLOAD_URL/></url>
</file_info>
Line 172: Line 198:
= References =

 * [[http://boinc.berkeley.edu/trac/wiki/JobSubmission|JobSubmission]] entry on BOINC Wiki
 * [[http://boinc.berkeley.edu/trac/wiki/AppVersionNew|AppVersionNew]] entry on BOINC Wiki
------

This page explains how to create units of work for one's self-run BOINC project.

1. Add a Work Unit

A work unit is a portion of the project's data that the project should be analysed with a particular application. Applications and data can be shared between work units. The following parts need to be known for the creation of a workunit in the database.

  • Input file(s)
  • Work Unit template
  • Result Template

1.1. Input files

Input files are what the application should be working on. Typically those vary for every work unit, like the range of numbers in which to search for primes. We will learn at some later stage that such inputs can be shared between work units, like a particular protein structure against which a large set of ligands shall be tested for their molecular interaction.

The example application this tutorial addresses converts files in mixed case to a file with all letters in upper case. For this, we manually create an invariant test input file. It is a one-lined arbitrary string, stored in $installroot/$fileprojectname/download/input_file in all lower case. The application run will convert it into all upper case.

. ~/.boinc_test.conf
if test -n "$installroot" -a -n "$fileprojectname" && cd $installroot/$fileprojectname; then
   sudo strace -o download/input_file -- strace -h > /dev/null && echo "[ok]" || echo "[failed]"
else
   echo "E: please check parameters in ~/.boinc_test.conf"
fi

1.2. Template files

Copy the default work unit and result template for upper case application to the templates folder.

# The paths are to be changed in the next version of package, so watch out.
sudo cp /usr/share/doc/boinc-server-maker/examples/upper_case_{result,wu} $installroot/$fileprojectname/templates/

That is just two files

$ ls templates/
upper_case_result  upper_case_wu

Those are XML formatted descriptions of how workunits and results shall be looking like. In deep theory, every result could well look different, say lets one expect larger or smaller result files, depending on what the input is. But typically one would just specify an upper limit of file sizes, e.g. to reduce the negative effect that an evil doer or an error in the application might otherwise have.

The files are not overly complicated. In their basic form, this means. There is a job sumission documentation on the BOINC Wiki indicating much more thoughts to possibly be put into it all:

$ cat /var/tmp/boinc-server-test/boinctest/templates/upper_case_result
<file_info>
  <name><OUTFILE_0/></name>
  <generated_locally/>
  <upload_when_present/>
  <max_nbytes>10000</max_nbytes>
  <url><UPLOAD_URL/></url>
</file_info>
<result>
  <file_ref>
    <file_name><OUTFILE_0/></file_name>
    <open_name>out</open_name>
  </file_ref>
</result>

and similarly

$ cat /var/tmp/boinc-server-test/boinctest/templates/upper_case_wu 
<file_info>
  <number>0</number>
</file_info>
<workunit>
  <file_ref>
    <file_number>0</file_number>
    <open_name>in</open_name>
  </file_ref>
  <min_quorum>1</min_quorum>
  <target_nresults>1</target_nresults>
</workunit>

which with some confidence is working for quite a number of different projects.

1.3. Fill the database

We will now introduce the system to a single work unit - only a single. The input remains described in the file input_file, templates are specified individually for result and work unit (wu). The application name is read from the command line, not from the wu description. This way, one could have different applications work on the same work unit more easily. Every unit is also passed a name, which the user sees in the BOINC manager and shall identify the work unit.

We will call the workunit test and to commit it to the database, execute

[ -n "$installroot" -a -n  "$fileprojectname" ] && \
cd $installroot/$fileprojectname && \
sudo ./bin/create_work -appname upper_case -wu_name test -wu_template templates/upper_case_wu -result_template templates/upper_case_result input_file || echo "E: something failed"

1.4. Uploading multiple work units

All input files need to be created at some stage. So they will be available, most likely as some file or they can be made available as a file. For batch invocations of create_work, one can use the shell's loop functionality. For the BASH, this could for instance be

for i in $(seq 1 3); do
  echo ./bin/create_work -appname upper_case -wu_name test$i input_${i}_file
done

yielding

./bin/create_work -appname upper_case -wu_name test1 input_1_file
./bin/create_work -appname upper_case -wu_name test2 input_2_file
./bin/create_work -appname upper_case -wu_name test3 input_3_file

1.5. Inspections (optional)

1.5.1. Files created

The download folder will now have a directory created especially for this workunit. To find this (or other modifications after the execution of some BOINC tool) seek the latest modified folder:

$ ls -ltr $installroot/$fileprojectname
...
drwxrwxr-x  2 root boincadm 4096 Jul  9 13:40 templates
drwxrwxr-x  3 root boincadm 4096 Jul  9 14:09 download

and in download this is

$ ls -ltr $installroot/$fileprojectname/download | tail -1
drwxrwx--x 2 root root   4096 Jul  9 14:09 76

which again shows

 ls -ltr $installroot/$fileprojectname/download/76
total 4
-rw-r--r-- 1 root root 12 Jul  9 14:09 input_file

which is the very same file.

1.5.2. Database inspection

mysql> select id,create_time,appid,name,fileset_id from workunit;
| id | create_time | appid | name | fileset_id |
+----+-------------+-------+------+------------+
|  1 |  1310213354 |    11 | test |          0 |

2. Multifile Applications

2.1. Multiple Binaries

With the renewed application-version-platform hierarchy outlined earlier, applications that consists of more than one file are already covered. We will get back to it in the context of legacy applications being wrapped for BOINC compatibility, which needs two apps - the wrapper and the legacy one. Please skim through ?BOINC/WrapperApp to mentally prepare for it.

2.2. Multiple Input/Output Files

The input/output files are specified in the workunit and result templates respectively. The <file_info>, <file_ref> tag pairs each describes a file. For example, if we need to create a workunit for test application which takes in1, in2 as input files, the workunit template would be

$ cat templates/test_wu
<file_info>
  <number>0</number>
</file_info>
<file_info>
  <number>1</number>
</file_info>

<workunit>
  <file_ref>
    <file_number>0</file_number>
    <open_name>in1</open_name>
  </file_ref>
  <file_ref>
    <file_number>1</file_number>
    <open_name>in2</open_name>
  </file_ref>
</workunit>

Please see each file has a <file_info> and <file_ref> tag. The same tags are used in case of result template,i.e if our application generates two output files, out1, out2 which should be transported back, The result template would be

$ cat template/test_result

<file_info>
  <name><OUTFILE_0/></name>
  <generated_locally/>
  <upload_when_present/>
  <max_nbytes>10000</max_nbytes>
  <url><UPLOAD_URL/></url>
</file_info>
<file_info>
  <name><OUTFILE_1/></name>
  <generated_locally/>
  <upload_when_present/>
  <max_nbytes>10000</max_nbytes>
  <url><UPLOAD_URL/></url>
</file_info>
<result>
  <file_ref>
    <file_name><OUTFILE_0/></file_name>
    <open_name>out1</open_name>
  </file_ref>
  <file_ref>
    <file_name><OUTFILE_1/></file_name>
    <open_name>out2</open_name>
  </file_ref>
</result>

3. References


Back to BOINC/ServerGuide