18640
Comment: fixed typo
|
15272
Fixing directory structure.
|
Deletions are marked like this. | Additions are marked like this. |
Line 8: | Line 8: |
''' This page is about deploying a test application with BOINC on Debian or Ubuntu. It extends the page [[BOINC/ServerGuide]] page.''' = Add scientific applications to be distributed = The BOINC project managers need to provide all the binaries for all the supported platforms. This is of some difficulty especially for those platforms that one does not own oneself. This page first demonstrates the workflow using xadd for a single platform. The second half of this page is dedicated to employing the binaries Debian provides for the purpose. The page [[BOINC/WrapperApp]] describes how to perform the equivalent installation for the [[http://boinc.berkeley.edu/trac/wiki/WrapperApp|Wrapper]] application. |
This page explains how to configure the BOINC server side to support and distribute a particular set of applications to its volunteers. = How to add a scientific application to a BOINC server = The BOINC project managers need to provide all the binaries for all the supported platforms. This is of some difficulty especially for those platforms that one does not own oneself. This page's first half demonstrates the workflow using xadd for a single platform, i.e. the one we can work with locally. The second half of this page is dedicated to employing the binaries Debian provides for the purpose. Debian supports many platforms, and we just reply on Debian's infrastructure to have built those correctly without our intervention. With BOINC's wrapper application, every regular command line execution can be run by BOINC. One just needs to distribute the wrapper together with the application. A separate [[BOINC/ServerGuide/WrapperApp|page]] describes how to substitute the BOINC-prepared example application described here with a triplet of the BOINC-prepared Wrapper application, a description on the invocation and the binary to run. And again, while input data and the invocation are platform agnostic, the platform-dependent applications and specialised libraries can be taken from Debian. This holds for all BOINC projects, not just for those that use the Debian boinc-server-maker package to get started. |
Line 18: | Line 18: |
The process is as follows. Within the project folder we have all binaries for all platforms for a particular application together. What we have we write into a file named "project.xml". Once done, we invoke the tool "xadd" from the project's bin folder. Once done, the BOINC server is ready to accept workunits for that application. |
|
Line 20: | Line 22: |
Install the application package, boinc-app-examples: {{{ apt-get install boinc-app-examples }}} The example application used in the document is ''upper_case'', which converts text inside a file to all capitals. Have a quick look that it is truly contained, since this walk-through may already be outdated (we are all volunteers): {{{ |
Install the example application package with the distribution's regular apt-get tool: {{{ $ sudo apt-get install boinc-app-examples }}} The example application used in the document is ''upper_case'', which converts text inside a file to all capitals. To verify that the package just installed truly shows that binary, it may just have been renamed, use regular Debian command line tools: {{{ |
Line 31: | Line 33: |
This directory becomes an intrinsic part of your project. {{{ [ -z "$installroot" -o -z "$fileprojectname" ] || . ~/.boinc_test.conf appdir="$installroot"/"$fileprojectname"/apps/upper_case sudo mkdir -p "$appdir" }}} Copy the file from the installed "boinc-app-examples" Debian package into that directory and rename it to distinguish versions and architectures. In our case, the ''app_ver'' variable is that of the BOINC server, the second part of the filename is that of the BOINC architecture. {{{ appver=6.12 # adjust to the right version, only have single "." |
This directory becomes an intrinsic part of your project. It needs to reside in a subdirectory of what is accessible through your project's website. After all, it is that website that is contacted by the BOINC clients. {{{ [ -z "$installroot" -o -z "$projectname" ] && . ~/.boinc_test.conf if test -z "$installroot" -o -z "$projectname"; then echo "E: ~/.boinc_test.conf does not have all parameters." else appdir="$installroot"/"$projectname"/apps/upper_case echo "I: The application directory is '$appdir'." sudo mkdir -p "$appdir" fi }}} We now copy the file from the above installed "boinc-app-examples" Debian package into that directory and rename it to distinguish versions and architectures. The naming obeys a particular theme that BOINC expects. In our case, the ''app_ver'' variable is that of the BOINC server, the second part of the filename is that of the BOINC architecture. The code below executes the tool 'arch' of the coreutils package to learn about the UNIX platform. The BOINC platforms commonly directly derive from them. The $( .. ) is equivalent to the backticks {{{` `}}} that may be more familiar. And when substituting a variable within a larger string, the variable name is enclosed in curly brackets ${...} . {{{ appver=7.042 # adjust to the right version, only have single "." |
Line 44: | Line 51: |
sudo cp $(dpkg -L boinc-app-examples | grep upper_case) $appdir/upper_case_${appver}_${boincplat} }}} Upstream lists official BOINC architectures [[http://boinc.berkeley.edu/trac/wiki/BoincPlatforms|here]]. Please keep the version formatted that simply - or change the BOINC source code. == Use the Debian-provided script to install binaries for multiple platforms == When applications do not have dependencies non-standard dynamically loaded libraries (test with the tool 'ldd'), then one can use the regular binary from Debian. This should then be functional also for non-Debian/Ubuntu platforms. The boinc-server-maker package provides a shell script that downloads the Debian packages of a given name (the default is the boinc-app-examples package) and unpacks, organizes and signs the binaries readily to be used with boinc-server. === Usage === 1. Obtain the script from /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh {{{ |
binary=$(dpkg -L boinc-app-examples | grep upper_case) echo "I: Application version: $appver" echo "I: BOINC platform: $boincplat" echo "I: binary at: $binary" if test -z "$appdir" -o -z "$appver" -o -z "$boincplat" -o -z "$binary"; then echo "E: Lost onee of the variables appdir, appver, binary or boincplat - please check and try again" else completepath=$appdir/$appver/$boincplat && \ sudo mkdir -p $completepath && \ sudo cp $binary $completepath/upper_case_${appver}_${boincplat} && \ echo "[ok]" fi }}} Upstream lists official BOINC architectures [[http://boinc.berkeley.edu/trac/wiki/BoincPlatforms|here]]. The hierarchy of directories looks like an overkill at a first sight. But over time, BOINC projects have become increasingly complex and more often than not, one wants to add more than a single binary. And that team of binaries and data files may have version dependencies among themselves. The official description of that format is on the BOINC wiki for [[http://boinc.berkeley.edu/trac/wiki/AppVersionNew|AppVersionNew]] - just slighty hidden. == Optional and outdated: Use the Debian-provided script to install binaries for multiple platforms == ''Some volunteer please bring the script up to speed for the new hieararchical directory structure as explained above. Until that is done, please bear with us and prepare those platform directories manually.'' When applications do not have dependencies on non-standard dynamically loaded libraries (test with the tool 'ldd'), then one can use the regular binary from Debian. This should then be functional also for non-Debian/Ubuntu platforms. The boinc-server-maker package provides a shell script that downloads the Debian packages of a given name (the default is the boinc-app-examples package) and unpacks, organizes and signs the binaries readily to be redistributed by the BOINC server. 1. Obtain the script from /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh {{{ |
Line 58: | Line 76: |
$ cp /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh . }}} 2. Edit the script to change $projectroot, and run. |
$ zcat /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh.gz > fetch_example_applications.sh }}} 1. Edit the script's source to set $projectroot, and run. |
Line 84: | Line 101: |
| |-- concat_6.12_armel-linux-gnu | |-- concat_6.12_armel-linux-gnu.sig . . [ i686, ia64, mips, s390, sparc not shown] . | |-- concat_6.12_x86_64-pc-linux-gnu | `-- concat_6.12_x86_64-pc-linux-gnu.sig |-- sleeper | |-- sleeper_6.12_armel-linux-gnu | |-- sleeper_6.12_armel-linux-gnu.sig . . [ i686, ia64, mips, s390, sparc not shown] . | |-- sleeper_6.12_x86_64-pc-linux-gnu | `-- sleeper_6.12_x86_64-pc-linux-gnu.sig |-- uc2 | |-- uc2_6.12_armel-linux-gnu | |-- uc2_6.12_armel-linux-gnu.sig . . [ i686, ia64, mips, s390, sparc ] . | |-- uc2_6.12_x86_64-pc-linux-gnu | `-- uc2_6.12_x86_64-pc-linux-gnu.sig |
... |
Line 116: | Line 111: |
| |-- worker_6.12_armel-linux-gnu | |-- worker_6.12_armel-linux-gnu.sig . . [ i686, ia64, mips, s390, sparc ] . | |-- worker_6.12_x86_64-pc-linux-gnu | `-- worker_6.12_x86_64-pc-linux-gnu.sig |
... |
Line 132: | Line 121: |
Not shown are the analogous entries for the example applications concat, uc2, sleeper and worker. | |
Line 136: | Line 126: |
cp ~/fetch-app/apps/upper_case $installroot/$fileprojectname/apps/ | cp ~/fetch-app/apps/upper_case/* $installroot/$fileprojectname/apps/upper_case/ |
Line 147: | Line 137: |
The project.xml file informs the BOINC server about what this project is all about, i.e. what platforms we are supporting with what tools (''apps''). You can copy and paste the following BASH shell lines to your local console. It will create the file ''project.xml'' in the project's root directory or bail out if something goes the unexpected way. {{{ [ -z "$installroot" -o -z "$fileprojectname" ] || . ~/.boinc_test.conf if [ -z "$installroot" -o -z "$fileprojectname" ]; then echo 'Variables $installroot (' $installroot ') and $fileprojectname (' $fileprojectname ') are both required.' elif [ -d "$installroot/$fileprojectname" ]; then |
The project.xml file informs the BOINC server about what this project is all about, i.e. what platforms we are supporting with what tools (''apps''). You can copy and paste the following BASH shell lines to your local console. It will create the file ''project.xml'' in the project's root directory or bail out if something goes the unexpected way. {{{ [ -z "$installroot" -o -z "$projectname" ] || . ~/.boinc_test.conf if [ -z "$installroot" -o -z "$projectname" ]; then echo 'Variables $installroot (' $installroot ') and $projectname (' $projectname ') are both required.' elif [ -d "$installroot/$projectname" ]; then if [ -f "$installroot/$projectname"/project.xml ]; then sudo gzip -9 --suffix ".$(date +"%Y%m%dT%H%M").gz" "$installroot/$projectname"/project.xml fi |
Line 167: | Line 155: |
<user_friendly_name>Linux/x86</user_friendly_name> | <user_friendly_name>Linux/x86 32bit</user_friendly_name> |
Line 171: | Line 159: |
<user_friendly_name>Linux/amd64</user_friendly_name> | <user_friendly_name>Linux/amd64 64bit</user_friendly_name> |
Line 175: | Line 163: |
) | sudo tee "$installroot"/"$fileprojectname"/project.xml fi }}} ''FIXME: the following paragraph is difficult to read and partially wrong, I think. From my observation, all platforms need to be specified manually. Nothing is copied from some sort of a reference project.xml.'' The platforms need not be defined here, since the default ''project.xml'' (found in /usr/share/boinc-server/tools/) contains entries for all platforms. They are shown here for the user to understand finer details. Duplicate entries on database will generate errors. In most cases, you can just append the application entries to the default ''project.xml'' file and run xadd. Change to the projectroot {{{ cd "$installroot"/"$fileprojectname" |
) | sudo tee "$installroot"/"$projectname"/project.xml fi }}} This file feed the xadd tool, which in turn feeds the project's MySQL database. The entries say that those applications should be prepared for to be eventually found. Thus, it is OK if only one of the 32 and 64 bit platforms are covered. Please look into standard platform names BOINC follows [[http://boinc.berkeley.edu/trac/wiki/BoincPlatforms|here]] to avoid confusion. Now let us invoke xadd. Change to the '''$projectroot''' {{{ cd "$installroot"/"$projectname" |
Line 191: | Line 177: |
Processing <Platform#None i686-pc-linux-gnu> ... Committed <Platform#3 i686-pc-linux-gnu> ; values: {'_dirty': False, '_lazy_lookups': {}, 'create_time': 1308988632L, 'deprecated': 0, 'id': 3L, 'name': 'i686-pc-linux-gnu', 'user_friendly_name': 'Linux running on an Intel x86-compatible CPU'} |
$ sudo bin/xadd |
Line 202: | Line 179: |
/var/tmp/boinc/boinctest/py/Boinc/db_base.py:63: Warning: Field 'host_scale_check' doesn't have a default value cursor.execute(command) Committed <App#11 upper_case> ; values: |
Committed <App#1 upper_case> ; values: |
Line 208: | Line 183: |
'create_time': 1309737828L, | 'create_time': 1357056725L, |
Line 210: | Line 185: |
'homogeneous_app_version': 0, | |
Line 212: | Line 188: |
'id': 11L, | 'id': 1L, 'locality_scheduling': 0L, |
Line 216: | Line 193: |
'non_cpu_intensive': 0, | |
Line 219: | Line 197: |
}}} This is the output of xadd parsing a single platform specification and a single application, the actual output is much longer due the increased number of platforms. Also it should be noted that currently xadd has no provision to delete from databases, it always appends the entries to databse, if you want to remove/change existing entries, you should do it manually. |
Processing <Platform#None i686-pc-linux-gnu> ... Committed <Platform#1 i686-pc-linux-gnu> ; values: {'_dirty': False, '_lazy_lookups': {}, 'create_time': 1357056725L, 'deprecated': 0, 'id': 1L, 'name': 'i686-pc-linux-gnu', 'user_friendly_name': 'Linux/x86 32bit'} Processing <Platform#None x86_64-pc-linux-gnu> ... Committed <Platform#2 x86_64-pc-linux-gnu> ; values: {'_dirty': False, '_lazy_lookups': {}, 'create_time': 1357056725L, 'deprecated': 0, 'id': 2L, 'name': 'x86_64-pc-linux-gnu', 'user_friendly_name': 'Linux/amd64 64bit'} }}} This is the output of xadd parsing two platform specifications and a single application. More platforms with more applications make longer outputs. Also it should be noted that currently xadd has no provision to delete from databases. It always appends the entries to database, if you want to remove/change existing entries, you should do it manually. Please indicate any better solution or send us emails. |
Line 232: | Line 229: |
BOINC need to sign the application binaries before dispatch for security reasons. | BOINC signs the application binaries for security reasons. This way, the volunteer's client may rest somewhat assured that the application's binary was not modified by someone with evil intentions, say, on the way from the project's server to the local machine. |
Line 241: | Line 238: |
sudo ./bin/sign_executable apps/upper_case/upper_case_${appver}_${boincplat} "$privateKeyfile" | sudo tee apps/upper_case/upper_case_${appver}_${boincplat}.sig | sudo ./bin/sign_executable apps/upper_case/$appver/$boincplat/upper_case_${appver}_${boincplat} "$privateKeyfile" | sudo tee apps/upper_case/$appver/$boincplat/upper_case_${appver}_${boincplat}.sig |
Line 247: | Line 244: |
./bin/update_versions | sudo ./bin/update_versions |
Line 254: | Line 251: |
Toshiba:/var/tmp/boinc/boinctest# ./bin/update_versions Found <App#11 upper_case> version 612 for <Platform#2 x86_64-pc-linux-gnu>: upper_case_6.12_x86_64-pc-linux-gnu Using signature file /var/tmp/boinc/boinctest/apps/upper_case/upper_case_6.12_x86_64-pc-linux-gnu.sig Copying upper_case_6.12_x86_64-pc-linux-gnu to /var/tmp/boinc/boinctest/download/upper_case_6.12_x86_64-pc-linux-gnu Ready to commit 1 items: <AppVersion#None upper_case 612 x86_64-pc-linux-gnu> Continue [Y/n] y Committed: <AppVersion#1 upper_case 612 x86_64-pc-linux-gnu> Touched trigger file to make feeder re-read app_version table from database Done }}} It should be noted that the app directory is just a staging area for the script to parse the structure and put the binaries to respective places. After this, the app directory can be removed safely. |
moeller@twin1a:/var/tmp/boinc-server-test/boinctest $ sudo ./bin/update_versions Found application version: upper_case 7.042 x86_64-pc-linux-gnu PHP Warning: Creating default object from empty value in /var/tmp/boinc-server-test/boinctest/bin/update_versions on line 410 PHP Warning: Creating default object from empty value in /var/tmp/boinc-server-test/boinctest/bin/update_versions on line 230 cp apps/upper_case/7.042/x86_64-pc-linux-gnu/upper_case_7.42_x86_64-pc-linux-gnu /var/tmp/boinc-server-test/boinctest/download/upper_case_7.42_x86_64-pc-linux-gnu Files: upper_case_7.42_x86_64-pc-linux-gnu (main program) Do you want to add this application version (y/n)? y Application version added successfully; ID=1 }}} It should be noted that the app directory is just a staging area for the script to parse the structure and put the binaries to respective places, i.e. the "download" directory. After this, the app directory can be removed safely. |
Line 287: | Line 284: |
= Add a Work Unit = A work unit is the portion of data that the project should be analyzed. It has the following parts, * Input file(s) * Work Unit template * Result Template == Input files == Input files are what the application should be working on. But typically those vary for every work unit, like the range of numbers in which to search for primes. We will learn at some later stage that such inputs can be shared between work units, like a particular protein structure against which a large set of ligands shall be tested for their molecular interaction. For this start, we manually create an invariant test input file. It is a one-lined arbitrary string, stored in $installroot/$fileprojectname/download/input_file in all lower case. The application run will convert it into all upper case. {{{ cd $installroot/$fileprojectname && echo test string >> download/input_file || echo "Something went wrong, check variables." }}} == Template files == Copy the default work unit and result template for upper case application to the templates folder. {{{ # The paths are to be changed in the next version of package, so watch out. cp /usr/share/doc/boinc-server-maker/examples/upper_case_* $installroot/$fileprojectname/templates/ }}} That is just two files {{{ $ ls templates/ upper_case_result upper_case_wu }}} Those are XML formatted descriptions of how workunits and results shall be looking like. In deep theory, every result could well look different, say lets one expect larger or smaller result files, depending on what the input is. But typically one would just specify an upper limit of file sizes, e.g. to reduce the negative effect that an evil doer or an error in the application might otherwise have. == Fill the database == We will now introduce the system to a single work unit - only a single. The input remains described in the file ''input_file'', templates are specified individually for result and work unit (wu). The application name is read from the command line, not from the wu description. This way, one could have different applications work on the same work unit more easily. Every unit is also passed a name, which the user sees in the BOINC manager and shall identify the work unit. We will call the workunit test and to commit it to the database, execute {{{ cd $installroot/$fileprojectname && \ ./bin/create_work -appname upper_case -wu_name test -wu_template templates/upper_case_wu -result_template templates/upper_case_result input_file }}} == Uploading multiple inputs == ''FIXME: this should be "multiple workunits", not "multiple inputs, right?"'' All input files need to be created at some stage. So they will be available, most likely as some file or they can be made available as a file. For batch invocations of ''create_work'', one can use the shell's loop functionality. For the BASH, this could for instance be {{{ for i in $(seq 1 3); do echo "./bin/create_work -appname upper_case -wu_name test$i input_${i}_file" done }}} yielding{{{ ./bin/create_work -appname upper_case -wu_name test1 input_1_file ./bin/create_work -appname upper_case -wu_name test2 input_2_file ./bin/create_work -appname upper_case -wu_name test3 input_3_file }}} == Inspections (optional) == === Files created === The download folder will now have a directory created especially for this workunit. To find this (or other modifications after the execution of some BOINC tool) seek the latest modified folder: {{{ $ ls -ltr $installroot/$fileprojectname drwxrwxr-x 2 root boincadm 4096 Jul 9 13:40 templates drwxrwxr-x 3 root boincadm 4096 Jul 9 14:09 download }}} and in download this is {{{ $ ls -ltr $installroot/$fileprojectname/download | tail -1 drwxrwx--x 2 root root 4096 Jul 9 14:09 76 }}} which again shows {{{ ls -ltr $installroot/$fileprojectname/download/76 total 4 -rw-r--r-- 1 root root 12 Jul 9 14:09 input_file }}} which is the very same file. === Database inspection === {{{ mysql> select id,create_time,appid,name,fileset_id from workunit; | id | create_time | appid | name | fileset_id | +----+-------------+-------+------+------------+ | 1 | 1310213354 | 11 | test | 0 | }}} = Multifile Applications = == Multiple Binaries == The applications which consists of more than one files, should be arranged in the following hierarchy for them to be parsed by ''bin/update_versions'' {{{ apps/ `-- appname |-- appname_version_platformname1 | `-- FILES |-- appname_version_platformname2 `-- FILES }}} Then do {{{ $ bin/update_versions }}} to commit to the database. Please see [[BOINC/WrapperApp]] deployment page, which is a multifile BOINC application. == Multiple Input/Output Files == The input/output files are specified in the workunit and result templates respectively. The '''<file_info>''', '''<file_ref>''' tag pairs each describes a file. For example, if we need to create a workunit for ''test'' application which takes ''in1'', ''in2'' as input files, the workunit template would be {{{ $ cat templates/test_wu <file_info> <number>0</number> </file_info> <file_info> <number>1</number> </file_info> <workunit> <file_ref> <file_number>0</file_number> <open_name>in1</open_name> </file_ref> <file_ref> <file_number>1</file_number> <open_name>in2</open_name> </file_ref> </workunit> }}} Please see each file has a '''<file_info>''' and '''<file_ref>''' tag. The same tags are used in case of result template,i.e if our application generates two output files, ''out1'', ''out2'' which should be transported back, The result template would be {{{ $ cat template/test_result <file_info> <name><OUTFILE_0/></name> <generated_locally/> <upload_when_present/> <max_nbytes>10000</max_nbytes> <url><UPLOAD_URL/></url> </file_info> <file_info> <name><OUTFILE_1/></name> <generated_locally/> <upload_when_present/> <max_nbytes>10000</max_nbytes> <url><UPLOAD_URL/></url> </file_info> <result> <file_ref> <file_name><OUTFILE_0/></file_name> <open_name>out1</open_name> </file_ref> <file_ref> <file_name><OUTFILE_1/></file_name> <open_name>out2</open_name> </file_ref> </result> }}} Back to [[BOINC/ServerGuide]]. |
= References = * [[http://boinc.berkeley.edu/trac/wiki/XaddTool|xadd]] BOINC Wiki entry * [[http://boinc.berkeley.edu/trac/wiki/AppVersionNew|new app version hierarchy]] BOINC Wiki entry ----- Back to [[BOINC/ServerGuide]] |
Contents
This page explains how to configure the BOINC server side to support and distribute a particular set of applications to its volunteers.
1. How to add a scientific application to a BOINC server
The BOINC project managers need to provide all the binaries for all the supported platforms. This is of some difficulty especially for those platforms that one does not own oneself. This page's first half demonstrates the workflow using xadd for a single platform, i.e. the one we can work with locally. The second half of this page is dedicated to employing the binaries Debian provides for the purpose. Debian supports many platforms, and we just reply on Debian's infrastructure to have built those correctly without our intervention.
With BOINC's wrapper application, every regular command line execution can be run by BOINC. One just needs to distribute the wrapper together with the application. A separate page describes how to substitute the BOINC-prepared example application described here with a triplet of the BOINC-prepared Wrapper application, a description on the invocation and the binary to run. And again, while input data and the invocation are platform agnostic, the platform-dependent applications and specialised libraries can be taken from Debian. This holds for all BOINC projects, not just for those that use the Debian boinc-server-maker package to get started.
1.1. Add a single example app for a single architecture to the BOINC project
The process is as follows. Within the project folder we have all binaries for all platforms for a particular application together. What we have we write into a file named "project.xml". Once done, we invoke the tool "xadd" from the project's bin folder. Once done, the BOINC server is ready to accept workunits for that application.
1.1.1. Get binary of local platform
Install the example application package with the distribution's regular apt-get tool:
$ sudo apt-get install boinc-app-examples
The example application used in the document is upper_case, which converts text inside a file to all capitals. To verify that the package just installed truly shows that binary, it may just have been renamed, use regular Debian command line tools:
$ dpkg -L boinc-app-examples | grep upper_case /usr/lib/boinc-server/apps/upper_case
1.1.2. Create a directory and add the app to project configuration.
This directory becomes an intrinsic part of your project. It needs to reside in a subdirectory of what is accessible through your project's website. After all, it is that website that is contacted by the BOINC clients.
[ -z "$installroot" -o -z "$projectname" ] && . ~/.boinc_test.conf if test -z "$installroot" -o -z "$projectname"; then echo "E: ~/.boinc_test.conf does not have all parameters." else appdir="$installroot"/"$projectname"/apps/upper_case echo "I: The application directory is '$appdir'." sudo mkdir -p "$appdir" fi
We now copy the file from the above installed "boinc-app-examples" Debian package into that directory and rename it to distinguish versions and architectures. The naming obeys a particular theme that BOINC expects. In our case, the app_ver variable is that of the BOINC server, the second part of the filename is that of the BOINC architecture. The code below executes the tool 'arch' of the coreutils package to learn about the UNIX platform. The BOINC platforms commonly directly derive from them. The $( .. ) is equivalent to the backticks ` ` that may be more familiar. And when substituting a variable within a larger string, the variable name is enclosed in curly brackets ${...} .
appver=7.042 # adjust to the right version, only have single "." boincplat=$(arch)-pc-linux-gnu # adjust to your architecture, maybe i686-pc-linux-gnu binary=$(dpkg -L boinc-app-examples | grep upper_case) echo "I: Application version: $appver" echo "I: BOINC platform: $boincplat" echo "I: binary at: $binary" if test -z "$appdir" -o -z "$appver" -o -z "$boincplat" -o -z "$binary"; then echo "E: Lost onee of the variables appdir, appver, binary or boincplat - please check and try again" else completepath=$appdir/$appver/$boincplat && \ sudo mkdir -p $completepath && \ sudo cp $binary $completepath/upper_case_${appver}_${boincplat} && \ echo "[ok]" fi
Upstream lists official BOINC architectures here. The hierarchy of directories looks like an overkill at a first sight. But over time, BOINC projects have become increasingly complex and more often than not, one wants to add more than a single binary. And that team of binaries and data files may have version dependencies among themselves.
The official description of that format is on the BOINC wiki for AppVersionNew - just slighty hidden.
1.2. Optional and outdated: Use the Debian-provided script to install binaries for multiple platforms
Some volunteer please bring the script up to speed for the new hieararchical directory structure as explained above. Until that is done, please bear with us and prepare those platform directories manually.
When applications do not have dependencies on non-standard dynamically loaded libraries (test with the tool 'ldd'), then one can use the regular binary from Debian. This should then be functional also for non-Debian/Ubuntu platforms. The boinc-server-maker package provides a shell script that downloads the Debian packages of a given name (the default is the boinc-app-examples package) and unpacks, organizes and signs the binaries readily to be redistributed by the BOINC server.
Obtain the script from /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh
$ mkdir ~/fetch-app && cd ~/fetch-app $ zcat /usr/share/doc/boinc-server-maker/examples/fetch_example_applications.sh.gz > fetch_example_applications.sh
- Edit the script's source to set $projectroot, and run.
The application will now be downloaded, and the directory structure will look somewhat like this. The .sig files may already have been added by the fetch_example_applications.sh tool, if your environment variables suggest an active project. But that is optional.
$ tree ~/fetch-app/apps apps/ |-- 1sec | |-- 1sec_6.12_armel-linux-gnu | |-- 1sec_6.12_armel-linux-gnu.sig | |-- 1sec_6.12_i686-pc-linux-gnu | |-- 1sec_6.12_i686-pc-linux-gnu.sig | |-- 1sec_6.12_ia64-linux-gnu | |-- 1sec_6.12_ia64-linux-gnu.sig | |-- 1sec_6.12_mips-linux-gnu | |-- 1sec_6.12_mips-linux-gnu.sig | |-- 1sec_6.12_s390-linux-gnu | |-- 1sec_6.12_s390-linux-gnu.sig | |-- 1sec_6.12_sparc-linux-gnu | |-- 1sec_6.12_sparc-linux-gnu.sig | |-- 1sec_6.12_x86_64-pc-linux-gnu | `-- 1sec_6.12_x86_64-pc-linux-gnu.sig |-- concat ... |-- upper_case | |-- upper_case_6.12_armel-linux-gnu | |-- upper_case_6.12_armel-linux-gnu.sig . . [ i686, ia64, mips, s390, sparc not shown] . | |-- upper_case_6.12_x86_64-pc-linux-gnu | `-- upper_case_6.12_x86_64-pc-linux-gnu.sig |-- worker ... `-- wrapper |-- wrapper_6.12_armel-linux-gnu |-- wrapper_6.12_armel-linux-gnu.sig . . [ i686, ia64, mips, s390, sparc not shown] . |-- wrapper_6.12_x86_64-pc-linux-gnu `-- wrapper_6.12_x86_64-pc-linux-gnu.sig
Not shown are the analogous entries for the example applications concat, uc2, sleeper and worker.
Copy the upper_case app to the project.
$ . ~/.boinc_test.conf && \ cp ~/fetch-app/apps/upper_case/* $installroot/$fileprojectname/apps/upper_case/
The boinc_test.conf sets the variables $installroot etc. Start at BOINC/ServerGuide if your search engine had brought you here directly.
2. Inform local database of available binaries
2.1. Craft the project's project.xml file
The project.xml file informs the BOINC server about what this project is all about, i.e. what platforms we are supporting with what tools (apps). You can copy and paste the following BASH shell lines to your local console. It will create the file project.xml in the project's root directory or bail out if something goes the unexpected way.
[ -z "$installroot" -o -z "$projectname" ] || . ~/.boinc_test.conf if [ -z "$installroot" -o -z "$projectname" ]; then echo 'Variables $installroot (' $installroot ') and $projectname (' $projectname ') are both required.' elif [ -d "$installroot/$projectname" ]; then if [ -f "$installroot/$projectname"/project.xml ]; then sudo gzip -9 --suffix ".$(date +"%Y%m%dT%H%M").gz" "$installroot/$projectname"/project.xml fi (cat << EOPROJECTXML <boinc> <app> <name>upper_case</name> <user_friendly_name>upperCASE</user_friendly_name> </app> <platform> <name>i686-pc-linux-gnu</name> <user_friendly_name>Linux/x86 32bit</user_friendly_name> </platform> <platform> <name>x86_64-pc-linux-gnu</name> <user_friendly_name>Linux/amd64 64bit</user_friendly_name> </platform> </boinc> EOPROJECTXML ) | sudo tee "$installroot"/"$projectname"/project.xml fi
This file feed the xadd tool, which in turn feeds the project's MySQL database. The entries say that those applications should be prepared for to be eventually found. Thus, it is OK if only one of the 32 and 64 bit platforms are covered. Please look into standard platform names BOINC follows here to avoid confusion.
Now let us invoke xadd. Change to the $projectroot
cd "$installroot"/"$projectname"
and run initiate the addition of the binary found in the directory structure to the local database
sudo bin/xadd
The local screen output will be similar to
$ sudo bin/xadd Processing <App#None upper_case> ... Committed <App#1 upper_case> ; values: {'_dirty': False, '_lazy_lookups': {}, 'beta': 0, 'create_time': 1357056725L, 'deprecated': 0, 'homogeneous_app_version': 0, 'homogeneous_redundancy': 0, 'host_scale_check': 0, 'id': 1L, 'locality_scheduling': 0L, 'min_avg_pfc': 1.0, 'min_version': 0L, 'name': 'upper_case', 'non_cpu_intensive': 0, 'target_nresults': 0, 'user_friendly_name': 'upperCASE', 'weight': 1.0} Processing <Platform#None i686-pc-linux-gnu> ... Committed <Platform#1 i686-pc-linux-gnu> ; values: {'_dirty': False, '_lazy_lookups': {}, 'create_time': 1357056725L, 'deprecated': 0, 'id': 1L, 'name': 'i686-pc-linux-gnu', 'user_friendly_name': 'Linux/x86 32bit'} Processing <Platform#None x86_64-pc-linux-gnu> ... Committed <Platform#2 x86_64-pc-linux-gnu> ; values: {'_dirty': False, '_lazy_lookups': {}, 'create_time': 1357056725L, 'deprecated': 0, 'id': 2L, 'name': 'x86_64-pc-linux-gnu', 'user_friendly_name': 'Linux/amd64 64bit'}
This is the output of xadd parsing two platform specifications and a single application. More platforms with more applications make longer outputs. Also it should be noted that currently xadd has no provision to delete from databases. It always appends the entries to database, if you want to remove/change existing entries, you should do it manually. Please indicate any better solution or send us emails.
And when executing that line again, nothing happens since everything here is already inside database, :
# bin/xadd Processing <App#None upper_case> ... Skipped existing <App#None upper_case>
The file project.xml is not touched.
2.2. Sign the application binary
BOINC signs the application binaries for security reasons. This way, the volunteer's client may rest somewhat assured that the application's binary was not modified by someone with evil intentions, say, on the way from the project's server to the local machine.
privateKeyfile="./keys/code_sign_private" if [ -z "$appver" -o -z "$boincplat" ]; then echo "Please set appver and boincplat variables from above." elif [ ! -r "$privateKeyfile" ]; then echo 'Have your private key ready as created during setup, expected at $privateKeyfile .' else sudo ./bin/sign_executable apps/upper_case/$appver/$boincplat/upper_case_${appver}_${boincplat} "$privateKeyfile" | sudo tee apps/upper_case/$appver/$boincplat/upper_case_${appver}_${boincplat}.sig fi
Update the boinc database,
sudo ./bin/update_versions
and prompt yes when asked for confirmation.
Sample output:
moeller@twin1a:/var/tmp/boinc-server-test/boinctest $ sudo ./bin/update_versions Found application version: upper_case 7.042 x86_64-pc-linux-gnu PHP Warning: Creating default object from empty value in /var/tmp/boinc-server-test/boinctest/bin/update_versions on line 410 PHP Warning: Creating default object from empty value in /var/tmp/boinc-server-test/boinctest/bin/update_versions on line 230 cp apps/upper_case/7.042/x86_64-pc-linux-gnu/upper_case_7.42_x86_64-pc-linux-gnu /var/tmp/boinc-server-test/boinctest/download/upper_case_7.42_x86_64-pc-linux-gnu Files: upper_case_7.42_x86_64-pc-linux-gnu (main program) Do you want to add this application version (y/n)? y Application version added successfully; ID=1
It should be noted that the app directory is just a staging area for the script to parse the structure and put the binaries to respective places, i.e. the "download" directory. After this, the app directory can be removed safely.
2.3. Inspection of the database (optional)
The database has now seen a single application
$ echo "select * from app" | mysql -u boincadm -p$pw $dbprojectname Enter password: id create_time name min_version deprecated user_friendly_name homogeneous_redundancy weight beta target_nresults min_avg_pfc host_scale_check 1 1308465648 upper_case 0 0 upperCASE 0 1 0 0 1 0
on multiple platforms
$ echo "select * from platform;"| mysql -u boincadm -p$pw $dbprojectname Enter password: id create_time name user_friendly_name deprecated 1 1308465648 i686-pc-linux-gnu Linux running on an Intel x86-compatible CPU 0 2 1308465648 x86_64-pc-linux-gnu Linux running on an AMD x86_64 or Intel EM64T CPU 0
When you have the leisure, don't shy away from inspecting the database more. Except for app, app_version and platform, all tables are empty at this stage. Straight forward to learn, a very basic tutorial on MySQL will do as a preparation ... if required at all.
3. References
xadd BOINC Wiki entry
new app version hierarchy BOINC Wiki entry
Back to BOINC/ServerGuide