Differences between revisions 23 and 25 (spanning 2 versions)
Revision 23 as of 2011-07-10 19:11:24
Size: 17333
Editor: dhananjay
Comment: Usage instr: fetch_example_applications
Revision 25 as of 2011-07-23 05:05:17
Size: 17551
Editor: dhananjay
Comment:
Deletions are marked like this. Additions are marked like this.
Line 14: Line 14:
Another page describes how to perform the equivalent installation for the [[http://boinc.berkeley.edu/trac/wiki/WrapperApp|Wrapper]] application. The page [[BOINC/WrapperApp]] describes how to perform the equivalent installation for the [[http://boinc.berkeley.edu/trac/wiki/WrapperApp|Wrapper]] application.
Line 297: Line 297:
It should be noted that the app directory is just a staging area for the script to parse the structure and put the binaries to respective places. After this the app directory can be cleaned safely.

This page is about deploying a test application with BOINC on Debian or Ubuntu. It extends the page BOINC/ServerGuide page.

1. Add scientific applications to be distributed

The BOINC project managers need to provide all the binaries for all the supported platforms. This is of some difficulty especially for those platforms that one does not own oneself. This page first demonstrates the workflow using xadd for a single platform. The second half of this page is dedicated to employing the binaries Debian provides for the purpose.

The page ?BOINC/WrapperApp describes how to perform the equivalent installation for the Wrapper application.

1.1. Add a single example app for a single architecture to the BOINC project

1.1.1. Get binary of local platform

Install the application package, boinc-app-examples:

apt-get install boinc-app-examples

The example application used in the document is upper_case, which converts text inside a file to all capitals. Have a quick look that it is truly contained, since this walk-through may already be outdated (we are all volunteers):

$ dpkg -L boinc-app-examples | grep upper_case
/usr/lib/boinc-server/apps/upper_case

1.1.2. Create a directory and add the app to project configuration.

This directory becomes an intrinsic part of your project.

[ -z "$installroot" -o -z "$fileprojectname" ] || . ~/.boinc_test.conf
appdir="$installroot"/"$fileprojectname"/apps/upper_case
sudo mkdir -p "$appdir"

Copy the file from the installed "boinc-app-examples" Debian package into that directory and rename it to distinguish versions and architectures. In our case, the app_ver variable is that of the BOINC server, the second part of the filename is that of the BOINC architecture.

appver=6.12 # adjust to the right version, only have single "."
boincplat=$(arch)-pc-linux-gnu # adjust to your architecture, maybe i686-pc-linux-gnu
sudo cp $(dpkg -L boinc-app-examples | grep upper_case) $appdir/upper_case_${appver}_${boincplat}

Upstream lists official BOINC architectures here.

Please keep the version formatted that simply - or change the BOINC source code.

1.2. Use the Debian-provided script to install binaries for multiple platforms

When applications do not have dependencies non-standard dynamically loaded libraries (test with the tool 'ldd'), then one can use the regular binary from Debian. This should then be functional also for non-Debian/Ubuntu platforms. The boinc-server-maker package provides a shell script that downloads the Debian packages of a given name (the default is the boinc-app-examples package) and unpacks, organizes and signs the binaries readily to be used with boinc-server.

1.2.1. Usage

1. Obtain the script from /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh

$ mkdir ~/fetch-app && cd ~/fetch-app 
$ cp  /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh .

2. Edit the script to change $projectroot, and run.

The application will now be downloaded, and the driectory structure will look somewhat like this.

$ tree ~/fetch-app/apps
 
apps/
|-- 1sec
|   |-- 1sec_6.12_armel-linux-gnu
|   |-- 1sec_6.12_armel-linux-gnu.sig
|   |-- 1sec_6.12_i686-pc-linux-gnu
|   |-- 1sec_6.12_i686-pc-linux-gnu.sig
|   |-- 1sec_6.12_ia64-linux-gnu
|   |-- 1sec_6.12_ia64-linux-gnu.sig
|   |-- 1sec_6.12_mips-linux-gnu
|   |-- 1sec_6.12_mips-linux-gnu.sig
|   |-- 1sec_6.12_s390-linux-gnu
|   |-- 1sec_6.12_s390-linux-gnu.sig
|   |-- 1sec_6.12_sparc-linux-gnu
|   |-- 1sec_6.12_sparc-linux-gnu.sig
|   |-- 1sec_6.12_x86_64-pc-linux-gnu
|   `-- 1sec_6.12_x86_64-pc-linux-gnu.sig
|-- concat
|   |-- concat_6.12_armel-linux-gnu
|   |-- concat_6.12_armel-linux-gnu.sig
|   |-- concat_6.12_i686-pc-linux-gnu
|   |-- concat_6.12_i686-pc-linux-gnu.sig
|   |-- concat_6.12_ia64-linux-gnu
|   |-- concat_6.12_ia64-linux-gnu.sig
|   |-- concat_6.12_mips-linux-gnu
|   |-- concat_6.12_mips-linux-gnu.sig
|   |-- concat_6.12_s390-linux-gnu
|   |-- concat_6.12_s390-linux-gnu.sig
|   |-- concat_6.12_sparc-linux-gnu
|   |-- concat_6.12_sparc-linux-gnu.sig
|   |-- concat_6.12_x86_64-pc-linux-gnu
|   `-- concat_6.12_x86_64-pc-linux-gnu.sig
|-- sleeper
|   |-- sleeper_6.12_armel-linux-gnu
|   |-- sleeper_6.12_armel-linux-gnu.sig
|   |-- sleeper_6.12_i686-pc-linux-gnu
|   |-- sleeper_6.12_i686-pc-linux-gnu.sig
|   |-- sleeper_6.12_ia64-linux-gnu
|   |-- sleeper_6.12_ia64-linux-gnu.sig
|   |-- sleeper_6.12_mips-linux-gnu
|   |-- sleeper_6.12_mips-linux-gnu.sig
|   |-- sleeper_6.12_s390-linux-gnu
|   |-- sleeper_6.12_s390-linux-gnu.sig
|   |-- sleeper_6.12_sparc-linux-gnu
|   |-- sleeper_6.12_sparc-linux-gnu.sig
|   |-- sleeper_6.12_x86_64-pc-linux-gnu
|   `-- sleeper_6.12_x86_64-pc-linux-gnu.sig
|-- uc2
|   |-- uc2_6.12_armel-linux-gnu
|   |-- uc2_6.12_armel-linux-gnu.sig
|   |-- uc2_6.12_i686-pc-linux-gnu
|   |-- uc2_6.12_i686-pc-linux-gnu.sig
|   |-- uc2_6.12_ia64-linux-gnu
|   |-- uc2_6.12_ia64-linux-gnu.sig
|   |-- uc2_6.12_mips-linux-gnu
|   |-- uc2_6.12_mips-linux-gnu.sig
|   |-- uc2_6.12_s390-linux-gnu
|   |-- uc2_6.12_s390-linux-gnu.sig
|   |-- uc2_6.12_sparc-linux-gnu
|   |-- uc2_6.12_sparc-linux-gnu.sig
|   |-- uc2_6.12_x86_64-pc-linux-gnu
|   `-- uc2_6.12_x86_64-pc-linux-gnu.sig
|-- upper_case
|   |-- upper_case_6.12_armel-linux-gnu
|   |-- upper_case_6.12_armel-linux-gnu.sig
|   |-- upper_case_6.12_i686-pc-linux-gnu
|   |-- upper_case_6.12_i686-pc-linux-gnu.sig
|   |-- upper_case_6.12_ia64-linux-gnu
|   |-- upper_case_6.12_ia64-linux-gnu.sig
|   |-- upper_case_6.12_mips-linux-gnu
|   |-- upper_case_6.12_mips-linux-gnu.sig
|   |-- upper_case_6.12_s390-linux-gnu
|   |-- upper_case_6.12_s390-linux-gnu.sig
|   |-- upper_case_6.12_sparc-linux-gnu
|   |-- upper_case_6.12_sparc-linux-gnu.sig
|   |-- upper_case_6.12_x86_64-pc-linux-gnu
|   `-- upper_case_6.12_x86_64-pc-linux-gnu.sig
|-- worker
|   |-- worker_6.12_armel-linux-gnu
|   |-- worker_6.12_armel-linux-gnu.sig
|   |-- worker_6.12_i686-pc-linux-gnu
|   |-- worker_6.12_i686-pc-linux-gnu.sig
|   |-- worker_6.12_ia64-linux-gnu
|   |-- worker_6.12_ia64-linux-gnu.sig
|   |-- worker_6.12_mips-linux-gnu
|   |-- worker_6.12_mips-linux-gnu.sig
|   |-- worker_6.12_s390-linux-gnu
|   |-- worker_6.12_s390-linux-gnu.sig
|   |-- worker_6.12_sparc-linux-gnu
|   |-- worker_6.12_sparc-linux-gnu.sig
|   |-- worker_6.12_x86_64-pc-linux-gnu
|   `-- worker_6.12_x86_64-pc-linux-gnu.sig
`-- wrapper
    |-- wrapper_6.12_armel-linux-gnu
    |-- wrapper_6.12_armel-linux-gnu.sig
    |-- wrapper_6.12_i686-pc-linux-gnu
    |-- wrapper_6.12_i686-pc-linux-gnu.sig
    |-- wrapper_6.12_ia64-linux-gnu
    |-- wrapper_6.12_ia64-linux-gnu.sig
    |-- wrapper_6.12_mips-linux-gnu
    |-- wrapper_6.12_mips-linux-gnu.sig
    |-- wrapper_6.12_s390-linux-gnu
    |-- wrapper_6.12_s390-linux-gnu.sig
    |-- wrapper_6.12_sparc-linux-gnu
    |-- wrapper_6.12_sparc-linux-gnu.sig
    |-- wrapper_6.12_x86_64-pc-linux-gnu
    `-- wrapper_6.12_x86_64-pc-linux-gnu.sig

Copy the upper_case app to the project.

$ cp ~/fetch-app/apps/upper_case $projectroot/$fileprojectname/apps/

2. Inform local database of available binaries

2.1. Craft the project's project.xml file

Now add following lines to project.xml in the project root

[ -z "$installroot" -o -z "$fileprojectname" ] || . ~/.boinc_test.conf
if [ -z "$installroot" -o -z "$fileprojectname" ]; then
   echo 'Variables $installroot (' $installroot ') and $fileprojectname (' $fileprojectname ') are both required.'
elif [ -d "$installroot/$fileprojectname" ]; then
   (cat << EOPROJECTXML
<boinc>
 <app>
  <name>upper_case</name>
  <user_friendly_name>upperCASE</user_friendly_name>
 </app>
 <platform>
  <name>i686-pc-linux-gnu</name>
  <user_friendly_name>Linux/x86</user_friendly_name>
 </platform>
 <platform>
  <name>x86_64-pc-linux-gnu</name>
  <user_friendly_name>Linux/amd64</user_friendly_name>
 </platform>
</boinc>
EOPROJECTXML
) | sudo tee "$installroot"/"$fileprojectname"/project.xml
fi

The platforms need not be defined here, since the default project.xml contains entries for all platforms, They are shown here for the user to understand finer details, and having duplicate entries on database will generate errors. In most cases, you can just append the application entries to the default project.xml file (found in /usr/share/boinc-server/tools/) and run xadd.

Change to the projectroot

cd "$installroot"/"$fileprojectname"

and run initiate the addition of the binary found in the directory structure to the local database

sudo bin/xadd

The local screen output will be similar to

Processing <Platform#None i686-pc-linux-gnu> ...
  Committed <Platform#3 i686-pc-linux-gnu> ; values:
{'_dirty': False,
 '_lazy_lookups': {},
 'create_time': 1308988632L,
 'deprecated': 0,
 'id': 3L,
 'name': 'i686-pc-linux-gnu',
 'user_friendly_name': 'Linux running on an Intel x86-compatible CPU'}

Processing <App#None upper_case> ...
/var/tmp/boinc/boinctest/py/Boinc/db_base.py:63: Warning: Field 'host_scale_check' doesn't have a default value
  cursor.execute(command)
  Committed <App#11 upper_case> ; values:
{'_dirty': False,
 '_lazy_lookups': {},
 'beta': 0,
 'create_time': 1309737828L,
 'deprecated': 0,
 'homogeneous_redundancy': 0,
 'host_scale_check': 0,
 'id': 11L,
 'min_avg_pfc': 1.0,
 'min_version': 0L,
 'name': 'upper_case',
 'target_nresults': 0,
 'user_friendly_name': 'upperCASE',
 'weight': 1.0}

This is the output of xadd parsing a single platform specification and a single application, the actual output is much longer due the increased number of platforms. Also it should be noted that currently xadd has no provision to delete from databases, it always appends the entries to databse, if you want to remove/change existing entries, you should do it manually.

And when executing that line again, nothing happens since everything here is already inside database, :

# bin/xadd 
Processing <App#None upper_case> ...
  Skipped existing <App#None upper_case>

The file project.xml is not touched.

2.2. Sign the application binary

BOINC need to sign the application binaries before dispatch for security reasons.

privateKeyfile="./keys/code_sign_private"
if [ -z "$appver" -o -z "$boincplat" ]; then
   echo "Please set appver and boincplat variables from above."
elif [ ! -r "$privateKeyfile" ]; then
   echo 'Have your private key ready as created during setup, expected at $privateKeyfile .'
else 
   sudo ./bin/sign_executable apps/upper_case/upper_case_${appver}_${boincplat} "$privateKeyfile" | sudo tee apps/upper_case/upper_case_${appver}_${boincplat}.sig
fi

Update the boinc database,

./bin/update_versions

and prompt yes when asked for confirmation.

Sample output:

Toshiba:/var/tmp/boinc/boinctest# ./bin/update_versions 
  Found <App#11 upper_case> version 612 for <Platform#2 x86_64-pc-linux-gnu>: upper_case_6.12_x86_64-pc-linux-gnu
Using signature file /var/tmp/boinc/boinctest/apps/upper_case/upper_case_6.12_x86_64-pc-linux-gnu.sig
Copying upper_case_6.12_x86_64-pc-linux-gnu to /var/tmp/boinc/boinctest/download/upper_case_6.12_x86_64-pc-linux-gnu
Ready to commit 1 items:
    <AppVersion#None upper_case 612 x86_64-pc-linux-gnu>
Continue [Y/n]  y
Committed:
    <AppVersion#1 upper_case 612 x86_64-pc-linux-gnu>
Touched trigger file to make feeder re-read app_version table from database
Done

It should be noted that the app directory is just a staging area for the script to parse the structure and put the binaries to respective places. After this the app directory can be cleaned safely.

2.3. Inspection of the database (optional)

The database has now seen the application and platforms:

$ echo "select * from app" | mysql -u boincadm -p$pw $dbprojectname
Enter password: 
id      create_time     name    min_version     deprecated      user_friendly_name      homogeneous_redundancy  weight  beta    target_nresults min_avg_pfc     host_scale_check
1       1308465648      upper_case      0       0       upperCASE       0       1       0       0       1       0


$ echo "select * from platform;"| mysql -u boincadm -p$pw $dbprojectname
Enter password: 
id      create_time     name    user_friendly_name      deprecated
1       1308465648      i686-pc-linux-gnu       Linux running on an Intel x86-compatible CPU    0
2       1308465648      x86_64-pc-linux-gnu     Linux running on an AMD x86_64 or Intel EM64T CPU       0

When you have the leisure, don't shy away from inspecting the database more. Except for app, app_version and platform, all tables are empty at this stage. Straight forward to learn, a very basic tutorial on MySQL will do.

3. Add a Work Unit

A work unit is the portion of data that the project should be analyzed, a work unit has following parts,

  • Input file(s)
  • Work Unit template
  • Result Template

3.1. Input files

Input files are what the application should be working on. But typically those vary for every work unit, like the range of numbers in which to search for primes. We will learn at some later stage that such inputs can be shared between work units, like a particular protein structure against which a large set of ligands shall be tested for their molecular interaction.

For this start, we manually create an invariant test input file. It is a one-lined arbitrary string, stored in $installroot/$fileprojectname/download/input_file in all lower case. The application run will convert it into all upper case.

cd $installroot/$fileprojectname && echo test string >> download/input_file || echo "Something went wrong, check variables."

3.2. Template files

Copy the default work unit and result template for upper case application to the templates folder.

# The paths are to be changed in the next version of package, so watch out.
cp /usr/share/doc/boinc-server-maker/examples/upper_case_* $installroot/$fileprojectname/templates/

That is just two files

$ ls templates/
upper_case_result  upper_case_wu

Those are XML formatted descriptions of how workunits and results shall be looking like. In deep theory, every result could well look different, say lets one expect larger or smaller result files, depending on what the input is. But typically one would just specify an upper limit of file sizes, e.g. to reduce the negative effect that an evil doer or an error in the application might otherwise have.

3.3. Fill the database

We will now introduce the system to a single work unit - only a single. The input remains described in the file input_file, templates are specified individually for result and work unit (wu). The application name is read from the command line, not from the wu description. This way, one could have different applications work on the same work unit more easily. Every unit is also passed a name, which the user sees in the BOINC manager and shall identify the work unit.

We will call the workunit test and to commit it to the database, execute

cd $installroot/$fileprojectname && \
./bin/create_work -appname upper_case -wu_name test -wu_template templates/upper_case_wu -result_template templates/upper_case_result input_file

3.4. Uploading multiple inputs

All input files need to be created at some stage. So they will be available, most likely as some file or they can be made available as a file. For batch invocations of create_work, one can use the shell's loop functionality. For the BASH, this could for instance be

for i in $(seq 1 3); do
  echo "-appname upper_case -wu_name test$i input_${i}_file"
done

yielding

-appname upper_case -wu_name test1 input_1_file
-appname upper_case -wu_name test2 input_2_file
-appname upper_case -wu_name test3 input_3_file

3.5. Inspections (optional)

3.5.1. Files created

The download folder will now have a directory created especially for this workunit. To find this (or other modifications after the execution of some BOINC tool) seek the latest modified folder:

$ ls -ltr $installroot/$fileprojectname
drwxrwxr-x  2 root boincadm 4096 Jul  9 13:40 templates
drwxrwxr-x  3 root boincadm 4096 Jul  9 14:09 download

and in download this is

$ ls -ltr $installroot/$fileprojectname/download | tail -1
drwxrwx--x 2 root root   4096 Jul  9 14:09 76

which again shows

 ls -ltr $installroot/$fileprojectname/download/76
total 4
-rw-r--r-- 1 root root 12 Jul  9 14:09 input_file

which is the very same file.

3.5.2. Database inspection

mysql> select id,create_time,appid,name,fileset_id from workunit;
| id | create_time | appid | name | fileset_id |
+----+-------------+-------+------+------------+
|  1 |  1310213354 |    11 | test |          0 |

Back to BOINC/ServerGuide.