Differences between revisions 33 and 36 (spanning 3 versions)
Revision 33 as of 2012-12-31 13:42:38
Size: 11857
Comment: Moved workunit generation to separate page
Revision 36 as of 2013-01-01 16:04:00
Size: 13208
Comment: Completed overhault of application download.
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
''' This page is about deploying a test application with BOINC on Debian or Ubuntu. It extends the page [[BOINC/ServerGuide]].'''

= Add scientific applications to be distributed =

The BOINC project managers need to provide all the binaries for all the supported platforms. This is of some difficulty especially for those platforms that one does not own oneself. This page first demonstrates the workflow using xadd for a single platform. The second half of this page is dedicated to employing the binaries Debian provides for the purpose.

The page [[BOINC/ServerGuide/WrapperApp]] describes how to perform the equivalent installation for the [[http://boinc.berkeley.edu/trac/wiki/WrapperApp|Wrapper]] application.
This page explains how to configure the BOINC server side to support and distribute a particular set of applications to its volunteers.

= How to add a scientific application to a BOINC server =

The BOINC project managers need to provide all the binaries for all the supported platforms. This is of some difficulty especially for those platforms that one does not own oneself. This page's first half demonstrates the workflow using xadd for a single platform, i.e. the one we can work with locally. The second half of this page is dedicated to employing the binaries Debian provides for the purpose. Debian supports many platforms, and we just reply on Debian's infrastructure to have built those correctly without our intervention.

With BOINC's wrapper application, every regular command line execution can be run by BOINC. One just needs to distribute the wrapper together with the application. A separate [[BOINC/ServerGuide/WrapperApp|page]] describes how to substitute the BOINC-prepared example application described here with a triplet of the BOINC-prepared Wrapper application, a description on the invocation and the binary to run. And again, while input data and the invocation are platform agnostic, the platform-dependent applications and specialised libraries can be taken from Debian. This holds for all BOINC projects, not just for those that use the Debian boinc-server-maker package to get started.
Line 18: Line 18:
The process is as follows. Within the project folder we have all binaries for all platforms for a particular application together. What we have we write into a file named "project.xml". Once done, we invoke the tool "xadd" from the project's bin folder. Once done, the BOINC server is ready to accept workunits for that application.
Line 20: Line 22:
Install the application package, boinc-app-examples: {{{ Install the example application package with the distribution's regular apt-get tool: {{{
Line 24: Line 26:
The example application used in the document is ''upper_case'', which converts text inside a file to all capitals. Have a quick look that it is truly contained, since this walk-through may already be outdated (we are all volunteers): {{{ The example application used in the document is ''upper_case'', which converts text inside a file to all capitals. To verify that the package just installed truly shows that binary, it may just have been renamed, use regular Debian command line tools: {{{
Line 31: Line 33:
This directory becomes an intrinsic part of your project.

{{{
[ -z "$installroot" -o -z "$fileprojectname" ] || . ~/.boinc_test.conf
appdir="$installroot"/"$fileprojectname"/apps/upper_case
sudo mkdir -p "$appdir"
}}}

Copy the file from the installed "boinc-app-examples" Debian package into that directory and rename it to distinguish versions and architectures. In our case, the ''app_ver'' variable is that of the BOINC server, the second part of the filename is that of the BOINC architecture.

{{{
appver=6.12 # adjust to the right version, only have single "."
This directory becomes an intrinsic part of your project. It needs to reside in a subdirectory of what is accessible through your project's website. After all, it is that website that is contacted by the BOINC clients.

{{{
[ -z "$installroot" -o -z "$fileprojectname" ] && . ~/.boinc_test.conf
if test -z "$installroot" -o -z "$fileprojectname"; then
   echo "E: ~/.boinc_test.conf does not have all parameters."
else
   appdir="$installroot"/"$fileprojectname"/apps/upper_case
   echo "I: The application directory is '$appdir'."
   sudo mkdir -p "$appdir"
fi
}}}

We now copy the file from the above installed "boinc-app-examples" Debian package into that directory and rename it to distinguish versions and architectures. The naming obeys a particular theme that BOINC expects. In our case, the ''app_ver'' variable is that of the BOINC server, the second part of the filename is that of the BOINC architecture. The code below executes the tool 'arch' of the coreutils package to learn about the UNIX platform. The BOINC platforms commonly directly derive from them. The $( .. ) is equivalent to the backticks {{{` `}}} that may be more familiar. And when substituting a variable within a larger string, the variable name is enclosed in curly brackets ${...} .

{{{
appver=7.042 # adjust to the right version, only have single "."
Line 44: Line 51:
echo "I: Application version: $appver"
echo "I: BOINC platform: $boincplat"
Line 48: Line 57:
Please keep the version formatted that simply - or change the BOINC source code.

== Use the Debian-provided script to install binaries for multiple platforms ==

When applications do not have dependencies non-standard dynamically loaded libraries (test with the tool 'ldd'), then one can use the regular binary from Debian. This should then be functional also for non-Debian/Ubuntu platforms. The boinc-server-maker package provides a shell script that downloads the Debian packages of a given name (the default is the boinc-app-examples package) and unpacks, organizes and signs the binaries readily to be used with boinc-server.
Please keep the version formatted that simple - or change the BOINC source code.

=== Optional: Use the Debian-provided script to install binaries for multiple platforms ===

When applications do not have dependencies on non-standard dynamically loaded libraries (test with the tool 'ldd'), then one can use the regular binary from Debian. This should then be functional also for non-Debian/Ubuntu platforms. The boinc-server-maker package provides a shell script that downloads the Debian packages of a given name (the default is the boinc-app-examples package) and unpacks, organizes and signs the binaries readily to be redistributed by the BOINC server.
Line 56: Line 65:
$ cp /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh . $ zcat /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh.gz > fetch_example_applications.sh
Line 81: Line 90:
| |-- concat_6.12_armel-linux-gnu
| |-- concat_6.12_armel-linux-gnu.sig
.
. [ i686, ia64, mips, s390, sparc not shown]
.
| |-- concat_6.12_x86_64-pc-linux-gnu
| `-- concat_6.12_x86_64-pc-linux-gnu.sig
|-- sleeper
| |-- sleeper_6.12_armel-linux-gnu
| |-- sleeper_6.12_armel-linux-gnu.sig
.
. [ i686, ia64, mips, s390, sparc not shown]
.
| |-- sleeper_6.12_x86_64-pc-linux-gnu
| `-- sleeper_6.12_x86_64-pc-linux-gnu.sig
|-- uc2
| |-- uc2_6.12_armel-linux-gnu
| |-- uc2_6.12_armel-linux-gnu.sig
.
. [ i686, ia64, mips, s390, sparc ]
.
| |-- uc2_6.12_x86_64-pc-linux-gnu
| `-- uc2_6.12_x86_64-pc-linux-gnu.sig
...
Line 113: Line 100:
| |-- worker_6.12_armel-linux-gnu
| |-- worker_6.12_armel-linux-gnu.sig
.
. [ i686, ia64, mips, s390, sparc ]
.
| |-- worker_6.12_x86_64-pc-linux-gnu
| `-- worker_6.12_x86_64-pc-linux-gnu.sig
...
Line 129: Line 110:
Not shown are the analogous entries for the example applications concat, uc2, sleeper and worker.
Line 133: Line 115:
  cp ~/fetch-app/apps/upper_case $installroot/$fileprojectname/apps/   cp ~/fetch-app/apps/upper_case/* $installroot/$fileprojectname/apps/upper_case
Line 281: Line 263:
   = References =
 * [[http://boinc.berkeley.edu/trac/wiki/XaddTool|xadd]] BOINC Wiki entry

-----

This page explains how to configure the BOINC server side to support and distribute a particular set of applications to its volunteers.

1. How to add a scientific application to a BOINC server

The BOINC project managers need to provide all the binaries for all the supported platforms. This is of some difficulty especially for those platforms that one does not own oneself. This page's first half demonstrates the workflow using xadd for a single platform, i.e. the one we can work with locally. The second half of this page is dedicated to employing the binaries Debian provides for the purpose. Debian supports many platforms, and we just reply on Debian's infrastructure to have built those correctly without our intervention.

With BOINC's wrapper application, every regular command line execution can be run by BOINC. One just needs to distribute the wrapper together with the application. A separate page describes how to substitute the BOINC-prepared example application described here with a triplet of the BOINC-prepared Wrapper application, a description on the invocation and the binary to run. And again, while input data and the invocation are platform agnostic, the platform-dependent applications and specialised libraries can be taken from Debian. This holds for all BOINC projects, not just for those that use the Debian boinc-server-maker package to get started.

1.1. Add a single example app for a single architecture to the BOINC project

The process is as follows. Within the project folder we have all binaries for all platforms for a particular application together. What we have we write into a file named "project.xml". Once done, we invoke the tool "xadd" from the project's bin folder. Once done, the BOINC server is ready to accept workunits for that application.

1.1.1. Get binary of local platform

Install the example application package with the distribution's regular apt-get tool:

apt-get install boinc-app-examples

The example application used in the document is upper_case, which converts text inside a file to all capitals. To verify that the package just installed truly shows that binary, it may just have been renamed, use regular Debian command line tools:

$ dpkg -L boinc-app-examples | grep upper_case
/usr/lib/boinc-server/apps/upper_case

1.1.2. Create a directory and add the app to project configuration.

This directory becomes an intrinsic part of your project. It needs to reside in a subdirectory of what is accessible through your project's website. After all, it is that website that is contacted by the BOINC clients.

[ -z "$installroot" -o -z "$fileprojectname" ] && . ~/.boinc_test.conf
if  test -z "$installroot" -o -z "$fileprojectname"; then
   echo "E: ~/.boinc_test.conf does not have all parameters."
else 
   appdir="$installroot"/"$fileprojectname"/apps/upper_case
   echo "I: The application directory is '$appdir'."
   sudo mkdir -p "$appdir"
fi

We now copy the file from the above installed "boinc-app-examples" Debian package into that directory and rename it to distinguish versions and architectures. The naming obeys a particular theme that BOINC expects. In our case, the app_ver variable is that of the BOINC server, the second part of the filename is that of the BOINC architecture. The code below executes the tool 'arch' of the coreutils package to learn about the UNIX platform. The BOINC platforms commonly directly derive from them. The $( .. ) is equivalent to the backticks `  ` that may be more familiar. And when substituting a variable within a larger string, the variable name is enclosed in curly brackets ${...} .

appver=7.042 # adjust to the right version, only have single "."
boincplat=$(arch)-pc-linux-gnu # adjust to your architecture, maybe i686-pc-linux-gnu
echo "I: Application version: $appver"
echo "I: BOINC platform: $boincplat"
sudo cp $(dpkg -L boinc-app-examples | grep upper_case) $appdir/upper_case_${appver}_${boincplat}

Upstream lists official BOINC architectures here.

Please keep the version formatted that simple - or change the BOINC source code.

1.1.3. Optional: Use the Debian-provided script to install binaries for multiple platforms

When applications do not have dependencies on non-standard dynamically loaded libraries (test with the tool 'ldd'), then one can use the regular binary from Debian. This should then be functional also for non-Debian/Ubuntu platforms. The boinc-server-maker package provides a shell script that downloads the Debian packages of a given name (the default is the boinc-app-examples package) and unpacks, organizes and signs the binaries readily to be redistributed by the BOINC server.

  1. Obtain the script from /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh

    $ mkdir ~/fetch-app && cd ~/fetch-app 
    $ zcat  /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh.gz > fetch_example_applications.sh
  2. Edit the script's source to set $projectroot, and run.

The application will now be downloaded, and the directory structure will look somewhat like this. The .sig files may already have been added by the fetch_example_applications.sh tool, if your environment variables suggest an active project. But that is optional.

$ tree ~/fetch-app/apps
 
apps/
|-- 1sec
|   |-- 1sec_6.12_armel-linux-gnu
|   |-- 1sec_6.12_armel-linux-gnu.sig
|   |-- 1sec_6.12_i686-pc-linux-gnu
|   |-- 1sec_6.12_i686-pc-linux-gnu.sig
|   |-- 1sec_6.12_ia64-linux-gnu
|   |-- 1sec_6.12_ia64-linux-gnu.sig
|   |-- 1sec_6.12_mips-linux-gnu
|   |-- 1sec_6.12_mips-linux-gnu.sig
|   |-- 1sec_6.12_s390-linux-gnu
|   |-- 1sec_6.12_s390-linux-gnu.sig
|   |-- 1sec_6.12_sparc-linux-gnu
|   |-- 1sec_6.12_sparc-linux-gnu.sig
|   |-- 1sec_6.12_x86_64-pc-linux-gnu
|   `-- 1sec_6.12_x86_64-pc-linux-gnu.sig
|-- concat
...
|-- upper_case
|   |-- upper_case_6.12_armel-linux-gnu
|   |-- upper_case_6.12_armel-linux-gnu.sig
.
.   [ i686, ia64, mips, s390, sparc not shown]
.
|   |-- upper_case_6.12_x86_64-pc-linux-gnu
|   `-- upper_case_6.12_x86_64-pc-linux-gnu.sig
|-- worker
...
`-- wrapper
    |-- wrapper_6.12_armel-linux-gnu
    |-- wrapper_6.12_armel-linux-gnu.sig
.
.   [ i686, ia64, mips, s390, sparc not shown]
.
    |-- wrapper_6.12_x86_64-pc-linux-gnu
    `-- wrapper_6.12_x86_64-pc-linux-gnu.sig

Not shown are the analogous entries for the example applications concat, uc2, sleeper and worker.

Copy the upper_case app to the project.

$ . ~/.boinc_test.conf && \
  cp ~/fetch-app/apps/upper_case/* $installroot/$fileprojectname/apps/upper_case

The boinc_test.conf sets the variables $installroot etc. Start at BOINC/ServerGuide if your search engine had brought you here directly.

2. Inform local database of available binaries

2.1. Craft the project's project.xml file

The project.xml file informs the BOINC server about what this project is all about, i.e. what platforms we are supporting with what tools (apps). You can copy and paste the following BASH shell lines to your local console. It will create the file project.xml in the project's root directory or bail out if something goes the unexpected way.

[ -z "$installroot" -o -z "$fileprojectname" ] || . ~/.boinc_test.conf
if [ -z "$installroot" -o -z "$fileprojectname" ]; then
   echo 'Variables $installroot (' $installroot ') and $fileprojectname (' $fileprojectname ') are both required.'
elif [ -d "$installroot/$fileprojectname" ]; then
   (cat << EOPROJECTXML
<boinc>
 <app>
  <name>upper_case</name>
  <user_friendly_name>upperCASE</user_friendly_name>
 </app>
 <platform>
  <name>i686-pc-linux-gnu</name>
  <user_friendly_name>Linux/x86</user_friendly_name>
 </platform>
 <platform>
  <name>x86_64-pc-linux-gnu</name>
  <user_friendly_name>Linux/amd64</user_friendly_name>
 </platform>
</boinc>
EOPROJECTXML
) | sudo tee "$installroot"/"$fileprojectname"/project.xml
fi

Please look into standard platform names BOINC follows here to remove future confusion. Change to the $projectroot

cd "$installroot"/"$fileprojectname"

and run initiate the addition of the binary found in the directory structure to the local database

sudo bin/xadd

The local screen output will be similar to

Processing <Platform#None i686-pc-linux-gnu> ...
  Committed <Platform#3 i686-pc-linux-gnu> ; values:
{'_dirty': False,
 '_lazy_lookups': {},
 'create_time': 1308988632L,
 'deprecated': 0,
 'id': 3L,
 'name': 'i686-pc-linux-gnu',
 'user_friendly_name': 'Linux running on an Intel x86-compatible CPU'}

Processing <App#None upper_case> ...
/var/tmp/boinc/boinctest/py/Boinc/db_base.py:63: Warning: Field 'host_scale_check' doesn't have a default value
  cursor.execute(command)
  Committed <App#11 upper_case> ; values:
{'_dirty': False,
 '_lazy_lookups': {},
 'beta': 0,
 'create_time': 1309737828L,
 'deprecated': 0,
 'homogeneous_redundancy': 0,
 'host_scale_check': 0,
 'id': 11L,
 'min_avg_pfc': 1.0,
 'min_version': 0L,
 'name': 'upper_case',
 'target_nresults': 0,
 'user_friendly_name': 'upperCASE',
 'weight': 1.0}

This is the output of xadd parsing a single platform specification and a single application, the actual output is much longer due the increased number of platforms. Also it should be noted that currently xadd has no provision to delete from databases, it always appends the entries to databse, if you want to remove/change existing entries, you should do it manually.

And when executing that line again, nothing happens since everything here is already inside database, :

# bin/xadd 
Processing <App#None upper_case> ...
  Skipped existing <App#None upper_case>

The file project.xml is not touched.

2.2. Sign the application binary

BOINC need to sign the application binaries before dispatch for security reasons.

privateKeyfile="./keys/code_sign_private"
if [ -z "$appver" -o -z "$boincplat" ]; then
   echo "Please set appver and boincplat variables from above."
elif [ ! -r "$privateKeyfile" ]; then
   echo 'Have your private key ready as created during setup, expected at $privateKeyfile .'
else 
   sudo ./bin/sign_executable apps/upper_case/upper_case_${appver}_${boincplat} "$privateKeyfile" | sudo tee apps/upper_case/upper_case_${appver}_${boincplat}.sig
fi

Update the boinc database,

./bin/update_versions

and prompt yes when asked for confirmation.

Sample output:

Toshiba:/var/tmp/boinc/boinctest# ./bin/update_versions 
  Found <App#11 upper_case> version 612 for <Platform#2 x86_64-pc-linux-gnu>: upper_case_6.12_x86_64-pc-linux-gnu
Using signature file /var/tmp/boinc/boinctest/apps/upper_case/upper_case_6.12_x86_64-pc-linux-gnu.sig
Copying upper_case_6.12_x86_64-pc-linux-gnu to /var/tmp/boinc/boinctest/download/upper_case_6.12_x86_64-pc-linux-gnu
Ready to commit 1 items:
    <AppVersion#None upper_case 612 x86_64-pc-linux-gnu>
Continue [Y/n]  y
Committed:
    <AppVersion#1 upper_case 612 x86_64-pc-linux-gnu>
Touched trigger file to make feeder re-read app_version table from database
Done

It should be noted that the app directory is just a staging area for the script to parse the structure and put the binaries to respective places. After this, the app directory can be removed safely.

2.3. Inspection of the database (optional)

The database has now seen a single application

$ echo "select * from app" | mysql -u boincadm -p$pw $dbprojectname
Enter password: 
id      create_time     name    min_version     deprecated      user_friendly_name      homogeneous_redundancy  weight  beta    target_nresults min_avg_pfc     host_scale_check
1       1308465648      upper_case      0       0       upperCASE       0       1       0       0       1       0

on multiple platforms

$ echo "select * from platform;"| mysql -u boincadm -p$pw $dbprojectname
Enter password: 
id      create_time     name    user_friendly_name      deprecated
1       1308465648      i686-pc-linux-gnu       Linux running on an Intel x86-compatible CPU    0
2       1308465648      x86_64-pc-linux-gnu     Linux running on an AMD x86_64 or Intel EM64T CPU       0

When you have the leisure, don't shy away from inspecting the database more. Except for app, app_version and platform, all tables are empty at this stage. Straight forward to learn, a very basic tutorial on MySQL will do as a preparation ... if required at all.

3. References

  • xadd BOINC Wiki entry


Back to BOINC/ServerGuide