Differences between revisions 26 and 27
Revision 26 as of 2017-08-04 18:45:18
Size: 21876
Comment: Explicitly remind folks to source the file that was just created
Revision 27 as of 2018-04-10 11:54:11
Size: 22587
Editor: ?paviwiki
Comment: Added a comment on mysql collation issue and php issue
Deletions are marked like this. Additions are marked like this.
Line 388: Line 388:
=== Comments ===
I tested this guide on Debian stretch with experimental repos.
I faced an error ``mysql_exceptions.OperationalError: (1071, 'Specified key was too long; max key length is 767 bytes')``
The reason this happens was that the default config of mysqld is
{{{
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci

}}}
while the recommended settings for BOINC are

{{{
character-set-server=utf8
collation-server=utf8_general_ci
}}}
So the mysqld config should be changed.


Another issue was a PHP exception ``Call to undefined function simplexml_load_file() boinc``
This is simply solved by
{{{
apt-get install php7.0-mbstring php7.0-zip php7.0-xml
}}}

1. BOINC Server Setup: Project Initialisation and the Default Website

1.1. Preparation

There is nothing to prepare if you do not go for a real project. Just try with the machine at hand. We maintain the boinc-server-maker package in the "experimental" section of Debian. The reason for that is that we do not want our users to use code that is likely to be two years old and invite the public to that server. It would miss any security leak that has since been identified and made public. As for any package distributed in experimental, it only installs to Debian "testing" (Stretch at the time of writing) or Debian "unstable" because of its most modern dependencies.

For projects inviting the public prepare for

  • disk space
  • network security
  • extra IT-savvy eyeballs for all the unforeseen things that may happen and need quick reactions to get fixed

For the moment, the Debian package does not create its own boincadm (speak: admin) user. We suggest the following, your local admins may have different ideas - talk back to us:

sudo adduser --system --group boincadm --no-create-home
sudo usermod -aG boincadm www-data

1.2. Software: BOINC Server Template files

This section first explains the installation of the BOINC server template sources from the source tree and then the same for those using the Debian boinc-server-maker package.

Previously

Now

1.

Install BOINC server dependencies:
sudo apt-get install git build-essential apache2 php5 \
     mysql-server php5-gd php5-cli php5-mysql python-mysqldb \
     libtool automake autoconf pkg-config libmysql++-dev libssl-dev

1.

Make sure that the experimental repository is in /etc/apt/sources.list:
deb http://ftp.debian.org/debian experimental main

2.

Make sure that your source list is up-to-date:
sudo apt-get update

2.

Download BOINC source:
git clone git://boinc.berkeley.edu/boinc-v2.git boinc

3.

Install boinc-server-maker package just like any other:
sudo apt-get install -t experimental boinc-server-maker

3.

Compile BOINC:
cd boinc; ./_autosetup; ./configure --disable-client --disable-manager; make

The compilation of the upstream source tree is nothing like magic. Go for it if you can. If our packaging we have done the right way, then this tutorial is of value for you because of the complete identity of all downstream steps. We ship the package in the experimental section for two reasons. A first is that we do not have any particular routine in setting those servers up. The second is that developments of BOINC Server and Client sides are not always stable at the same time, thus having the server in experimental grants us some extra flexibility.

1.3. Project-specific configuration

The preparation of the final project from the BOINC server template is a manual process. That should be applied only once for every project and then for updates of the BOINC server sources. One could (in principle) prepare Debian packages for such project specific configurations. The boinc-server-maker package described on this page would then become a build dependency for the project. See BOINC/Server/Projects for current (early) attempts.

1.3.1. Create and config BOINC project

The BOINC server as compiled from the BOINC sources does not perform directly.

The following needs to be performed for every project, with or without the Debian BOINC server package. In the following we will work on a single BASH shell. This allows us to use mostly self-explanatory variables to help your local adaptation.

To help working consistently on multiple shells, or to get the settings when logging in again, we need a place to redirect the parameters of the project configuration. We do that only to help ourselves; the BOINC code will not use it. This approach is well accepted, e.g. for .bash_aliases.

1.3.1.1. Initialisation of the configuration

For a first start of a project create the file that the project parameters shall be stored in.

echo "boincserverconfig=~/.boinc_test.conf" > ~/.boinc_test.conf
chmod 600 ~/.boinc_test.conf

Please run through this file now to set the boincserverconfig variable we also need below

. ~/.boinc_test.conf

and whenever starting a shell, these settings should be set, so you may decide to have an account dedicated for your project and have the configration sourced immediately after the shell was started, e.g. for BASH like:

cat >> ~/.bashrc <<EOBASHRC
if [ -r "$boincserverconfig" ]; then
   . $boincserverconfig
fi
EOBASHRC

1.3.1.2. Parameters for the MySQL database

cat << EODBCONFIG >> $boincserverconfig
# name of the MySQL database
dbname=boinctest
# name of the MySQL user with access to above database
dbuser=boincadm
# password for write access to your project database
dbpasswd=MYSQLPASSWORDFORBOINCUSER
EODBCONFIG

For your convenience you may copy above code and then use an editor to change it.

1.3.1.3. Create MySQL database for BOINC project

With the variables defined before, you can just copy and pasted the code below. The password asked is the password for the root user to access the database, which is not unlikely to differ from the root password to your UNIX system.

# read config if available
[ -r "$boincserverconfig" ] && . $boincserverconfig
if   [ -z "$dbuser" ]; then echo "Variable 'dbuser' not set";
elif ! echo "DROP USER '$dbuser'@'localhost'" | mysql -u root -p; then
   echo "If the removal of the previous user fails because the user is not existing, then this does not matter. Other errors please investigate."
fi

You are asked in the block above for a password and you will be again in the block below.

if   [ -z "$dbname" ]; then echo "Variable 'dbname' not set";
elif [ -z "$dbpasswd" ]; then echo "Variable 'dbpasswd' not set";
else
   # piping commands to mysql shell
   cat <<EOMYSQL | mysql -u root -p;
DROP DATABASE IF EXISTS $dbname;
CREATE USER '$dbuser'@'localhost' IDENTIFIED BY '$dbpasswd';
GRANT ALL PRIVILEGES ON $dbname.* TO '$dbuser'@'localhost';
EOMYSQL
fi

Over time, and if it is alone for backups, a basic familiarity with MySQL is (re|a)quired for the maintenance of BOINC projects.

1.3.2. Create project

Please replace a.b.c.d with your full hostname or your server's IP address. If you only have dynamic IP addresses, then this is unfortunate, but investigate the service of dyndns.org to help out.

cat <<EOCONF >> $boincserverconfig
# address of host (via DNS or IP number) at which project server shall be reached
hosturl=http://a.b.c.d
# name of folder in which data shall be stored, also becomes part of project URL
projectname=boinctest
# more human-compatible way to read the project name
projectnicename="BoincTestProject@Home"
# location to store the project files, a projectname subdirectory will be created
installroot=/var/lib/boinc-server-test
EOCONF

Again, please edit '~/.boinc_test.conf' with a text editor if required. This example installroot is where you cannot harm anything. Some linux distributions may however have that directory cleaned at boot time. Please adjust for permanent location.

So, to make sure you have indeed sourced the configuration paramters:

if [ ! -r "$boincserverconfig" ]; then
  echo "Configuration file '$boincserverconfig' not existing."
else
  . $boincserverconfig
  echo -n "Basic configuration test: "
  if [ -z "$installroot" -o -z "$hosturl" -o -z "$dbname" -o -z "$dbpasswd" \
    -o -z "$projectnicename" -o -z "$projectname" ] ; then
     echo "Missing configuration parameter."
  else
     echo "[ok]"
  fi
fi

Previously

Now

1.3.2.1. Installation according to the books - ''without'' the Debian server package

The installation is expected to happen from within the BOINC source tree.

[ -d "$installroot" ] || sudo mkdir -p "$installroot"
sudo ./tools/make_project \
  --url_base "$hosturl" \
  --db_name "$dbname" \
  --db_user "$dbuser" \
  --db_passwd "$dbpasswd" \
  --drop_db_first  \
  --project_root /var/www/boinc/$projectname \
    "$projectname" "$projectnicename"

1.3.2.2. Installation ''with'' the Debian server package

If you use boinc-server-maker package you need to use this command instead:

sudo PYTHONPATH=$PYTHONPATH:/usr/share/python-support/boinc-server/Boinc/ \
 /usr/share/boinc-server/tools/make_project \
  --url_base "$hosturl" \
  --db_name "$dbname" \
  --db_user "$dbuser" \
  --db_passwd "$dbpasswd" \
  --drop_db_first  \
  --project_root "$installroot"/"$projectname" \
  --srcdir /usr/share/boinc-server-maker/ \
   "$projectname" "$projectnicename"

source

source

With the original source tree or the Debian package, the project configuration is basically identical. The specification of the 'PYTHONPATH' should not be required. But it is for the moment, leaving some room for optimisation.

The '--test_app' flag is omitted to help the understanding of the setup. Also, it requires the compilation of an example_app which are all distributed separately from the boinc-server-maker package in the boinc-app-examples package, but not in the location that the install scripts expects them.

The screen output should not look too different from

Creating project 'BoincTestProject@Home' (short name 'boinctest'):
   PROJECT_ROOT = /var/tmp/boinc/boinctest/
       URL_BASE = http://boincserver.dyndns.org/
  HTML_USER_URL = http://boincserver.dyndns.org/boinctest/
   HTML_OPS_URL = http://boincserver.dyndns.org/boinctest_ops/
        CGI_URL = http://boincserver.dyndns.org/boinctest_cgi/
        KEY_DIR = /var/tmp/boinc/boinctest/keys/
        DB_NAME = boinctest
        DB_HOST =

Delete /var/tmp/boinc/boinctest/? [y/N]  y
Deleting /var/tmp/boinc/boinctest/

Continue? [Y/n]
Setting up server: creating directories

Keys don't exist in /var/tmp/boinc/boinctest/keys/; generate them? [Y/n]
Setting up server files: generating keys
Setting up server files: copying files
Setting up database
Setting up server files: writing config files
Setting up server files: linking cgi programs
update_translations finished
Done installing default daemons.

Done creating project.  Please view
/var/tmp/boinc/boinctest/boinctest.readme
for important additional instructions.

That 'boinctest.readme' basically says what the next section is presenting. If running into issues of various sorts, you may be suggested to repeat and overwrite ("clobber") the previous install. Caveat: when adding "--delete_prev_inst --drop_db_first", make sure to add them somewhere in the middle, not just comfortably at the end. The last two arguments are interpreted in a special way.

1.3.3. Continue

The instructions below are the same for installs that originated with the original BOINC distibution and those that used boinc-server-maker.

1.3.3.1. Adjusting permissions for project directory

  1. Make sure the configuration variables are set

    [ -r $boincserverconfig ] && . $boincserverconfig
  2. Change files and directories permission:

    if [ -z "$installroot" -o -z "$projectname" ]; then
      echo "Not all variables are set for the configuration"
      echo "Error, do not continue."
    elif [ ! -d "$installroot"/"$projectname" ]; then
      echo "The directory '$installroot/'$projectname' is not existing"
      echo "Error, do not continue."
    else
      cd "$installroot"/"$projectname"
      sudo chown boincadm:boincadm  -R .
      sudo chmod g+w -R .
      sudo chmod 02770 -R upload html/cache html/inc html/languages html/languages/compiled html/user_profile
      hostname=`hostname`
      if [ -z "$hostname" ]; then 
        echo "Please specify the hostname"
      else
        sudo chgrp -R www-data log_"$hostname" upload
      fi
    fi

A successful execution will once ask for the user's sudo password but otherwise remain quiet.

If those permissions are sufficient depends on the user the the web service runs with. For Debian, running apache as user www-data, the include files need to be accessible. FIXME: How exactly this is best achieved, still needs some extra thinking. To circumvent all issues for the moment, run

if [ -d html/inc -a -d cgi-bin ]; then
  echo -n "html/inc: "; sudo chmod o+x html/inc && sudo chmod -R o+r html/inc && echo "[ok]" || echo "[failed]"
  echo -n "html/languages: "; sudo chmod o+x html/languages/ html/languages/compiled && echo "[ok]" || echo "[failed]"
else
  echo "You are not in your project directory"
fi

The "-r" is important to reach all the files since the directory cannot be read before the execution of "sudo".

1.3.3.2. Start of BOINC background programs ("daemons")

Now its time to start the project, assuming we are still inside the project root,

$./bin/start
Staying in ENABLED mode
Starting daemons
  Starting daemon: feeder -d 3
  Starting daemon: transitioner -d 3
  Starting daemon: file_deleter -d 3

If you receive the message "No daemons for this host found - check host name in config.xml", then do just that and set the X in <hostname>X</hostname> to what the UNIX command hostname returns on your machine.

And ./bin/status can be used to check status of the project.

$ ./bin/status
BOINC is ENABLED

DAEMON  pid  status      lockfile disabled  commandline
   1  16574  running     locked   no        feeder -d 3
   2  16576  running     locked   no        transitioner -d 3
   3  16579  running     locked   no        file_deleter -d 3

TASK last run period    next run lock file disabled commandline
   1    ?     24 hours     NOW   unlocked  yes       db_dump -d 2 -dump_spec ../db_dump_spec.xml
   2    ?     1 days       NOW   unlocked  yes       run_in_ops ./update_uotd.php
   3    ?     1 hour       NOW   unlocked  yes       run_in_ops ./update_forum_activities.php
   4    ?     7 days       NOW   unlocked  yes       update_stats -update_users -update_teams -update_hosts
   5    ?     24 hours     NOW   unlocked  yes       run_in_ops ./update_profile_pages.php
   6    ?     24 hours     NOW   unlocked  yes       run_in_ops ./team_import.php
   7    ?     24 hours     NOW   unlocked  yes       run_in_ops ./notify.php

1.3.3.3. Automated restart upon failure

One can restart and restart the BOINC project without any harm. If it is already running, no change is performed. If something had stopped functioning, for whatever reason (it is a bug, most likely of because of some rare and hence barely tested condition) then the failing tool should be restarted. Here is how to do it.

  1. Add project cronjob:

    sudo EDITOR=vim crontab -e

Please check the exact path of the 'start' script. Then add a line like the following in the editor that spawned by the 'crontab -e'

  • command above. You can set the editor to be started by the EDITOR environment variable. Adjust to your liking.

    */5 * * * * /var/tmp/boinc-server-test/boinctest/bin/start --cron

The above indicates a restart (in case of failure) every five minutes. Verify that the cron entry was indeed accepted

sudo crontab -l
  1. Config password for BOINC server administrative page:

    sudo htpasswd -c html/ops/.htpasswd USERNAME
  2. Config Apache to call BOINC server

    sudo cp ${projectname}.httpd.conf  /etc/apache2/sites-available/ \
    && sudo a2ensite  ${projectname}.httpd.conf \
    && sudo /etc/init.d/apache2 reload

1.4. Access web interface

1.4.1. Apache preparation: Ensuring PHP and cgi apache modules are enabled, not only installed

If you see a page but that looks weird, then PHP may not be executed on that page but the source of the file is shown directly. To enable it, perform

cd /etc/apache2/mods-enabled
sudo ln -s ../mods-available/php5.* .
sudo ln -s ../mods-available/cgi.* .
sudo /etc/init.d/apache2 restart

alternative to the above you may prefer to write from anywhere

sudo a2enmod cgi
sudo a2enmod php5
sudo /etc/init.d/apache2 restart

1.4.2. Have a look

boinc_test_project_freshly_generated.jpg

The web site should now be accessible. It should be available as an alias "$projectname", please look it up in "/etc/apache2/sites-enabled/${projectname}.httpd.conf". You will find the automated home page to need some further manual adjustments - which does not really encourage to update the server version too frequently, we presume.

If the home page does not show up then there may be some missing file or some bogus file permission. The inspection of "/var/log/apache/error.log" should bring sufficient insights to fix the issue. The image shows a URL with a ":8080", which indicates the browser to ask for the page at a different port than the default ":80". For doing so, add a virtual host configuration in above httpd.conf of your project. The Apache Documentation explains vividly how to do so.

1.4.3. Trouble shooting

1.4.3.1. Showing index.php by default

boinc_test_project_freshly_generated_status_page.jpg

With a typical Debian apache setting, the expected BOINC web interface only shows up after clicking on index.php. It may be different for your setup, just if your project homepage looks more like source code then edit /etc/apache2/sites-enabled/${filesprojectname}.httpd.conf as follows to show index.php files directly:

<Directory "...fileprojectname.html">
   Options Indexes FollowSymlinks MultiViews +ExecCGI
   ...not shown...
   # Adding these three lines
   <IfModule mod_dir.c>
       DirectoryIndex index.php
   </IfModule>
</Directory>

Perform sudo /etc/init.d/apache2 reload after the change. Once PHP is properly configured, all dynamic web pages, including for accessing the database and message boards will run. You can also get your own account, just the pages all look ugly and unpersonal, still.

1.4.3.2. Ensure compatibility with extra security of Apache 2.4

Apache gained some extra security by not allowing access to files outside /var/www in its default configuration. To overcome this, add the line

Require all granted

to both directory instructions at /etc/apache2/sites-enabled/sleep.httpd.conf. This should already be addressed by later versions of the boinc-server-maker package.

1.4.4. Substituting ProjectName in *.html and *.php

The web pages show whenever there should be the project name the string "REPLACE WITH PROJECT NAME". To fix this and other things like your public email address, edit the file html/project/project.inc with any text editor, as in

sudo vim html/project/project.inc

That include file is read by various other PHP files and amongst the above mentioned user-visible project name and email addresses also allows for extra decorations for headers or footers.

While a project develops, one is likely to perform many additional changes well beyond initial cosmetics. The project's home/start page is a common initial target. Amend it in './html/user/index.php' but wary of subsequent updates. How exactly this will all be dealt with for updating is not yet clear. For the moment it is just suggested to have a backup when installing the next version of BOINC and then compare the HTML references made from that page. There are several tools known for software development which may come to a rescue, like bazaar, mercurial or git, that may help with the orchestrations of updates that come from new upstream developments and local additions. But that is independent from the Debian packaging side.

1.4.5. Get up forums up (optional)

Now you have the user-interface of your BOINC server up: the PHP web pages. The subfolder 'html/ops' contains the website management tools that are not called by the user but demand access through that admin password set above. The .PHP files are executed by a little engine within the Apache web server to display the results, in reaction to the respective PATH being selected via the browser. An alternative, and easier to many, is to call the same .PHP files through the command line.

As an example may serve the initialisation of the news. The screen shot shows it, where there should be the latest project news, there is a mere suggestion, reading "No news forum. Run html/ops/create_forums.php". To follow that advice, on the command line

[ -r "$boincserverconfig" ] && . $boincserverconfig

if [ ! -r create_forums.php ]; then
    if [ -r html/ops/create_forums.php ]; then 
        cd html/ops
    elif [ -r  $installroot/$fileprojectname/html/ops/create_forums.php ]; then
        cd $installroot/$fileprojectname/html/ops
    else
        echo "Could not find create_forums.php"
    fi
fi

Then edit the file to adjust the forums you want to be created or just

sudo sed -i '/remove the die/d' create_forums.php

to get the default for a start. The 'sed' command (so nice and helpful, but initially awkward) edits the .php script by removing an early exit. I means: search for a string "remove the die" and then deletes that line. If avoiding the sed (stream editor) tool, do so manually with your editor.

The execution of 'create_forums.php' can then be performed on the command line by

php5 create_forums.php

The user interface is - from a technical point of view - now set. It remains the application side to be configured.

Proceed with AppDeployment to learn about collecting binaries from multiple platforms of Debian and configure BOINC to serve them.

1.4.6. Comments

I tested this guide on Debian stretch with experimental repos. I faced an error mysql_exceptions.?OperationalError: (1071, 'Specified key was too long; max key length is 767 bytes') The reason this happens was that the default config of mysqld is

character-set-server = utf8mb4 
collation-server = utf8mb4_general_ci 

while the recommended settings for BOINC are

character-set-server=utf8 
collation-server=utf8_general_ci

So the mysqld config should be changed.

Another issue was a PHP exception Call to undefined function simplexml_load_file() boinc This is simply solved by

apt-get install php7.0-mbstring php7.0-zip php7.0-xml

Back