Project no. IST-033576 XtreemOS Integrated Project B UILDING AND P ROMOTING A L INUX - BASED O PERATING S YSTEM TO S UPPORT V IRTUAL O RGANIZATIONS FOR N EXT G ENERATION G RIDS Prototype of the basic version of Linux-XOS D2.1.4 Due date of deliverable: November 30th , 2007 Actual submission date: December 10th ,2007 Start date of project: June 1st 2006 Type: Deliverable WP number: WP2.1 Task number: T2.1.3/T2.1.
Revision history: Version Date 0.1 03/10/07 0.2 31/10/07 0.3 31/10/07 0.4 05/11/07 0.5 06/11/07 0.6 07/11/07 0.7 08/11/07 Authors Yvon Jégou Pascal Le Métayer Yvon Jégou Yvon Jégou Yvon Jégou Haiyan Yu Pascal Le Métayer Institution INRIA INRIA INRIA INRIA INRIA ICT INRIA 0.8 0.9 13/11/07 27/11/07 Yvon Jégou Pascal Le Métayer INRIA INRIA 1.
D2.1.4 IST-033576 Executive Summary This document presents a prototype of the basic version of Linux-XOS, the XtreemOS flavor for a single PC. This prototype is a first implementation of system services described in deliverables D2.1.2 [1] and D2.1.3 [2]. Procedures to get, install, configure, and experiment with each component of the prototype are described. Links to documentation for users and developers are also provided.
IST-033576 D2.1.4 counts. The next implementation will focus on the enforcement of local policies against VO users, by incorporating new isolation and virtualization mechanisms in Linux.
D2.1.4 IST-033576 Contents 1 Introduction 5 2 Kernel Checkpointer 2.1 Prerequisites . . . . . . . . . . . . . 2.2 Installation . . . . . . . . . . . . . 2.3 Tutorial/Command line tools . . . . 2.4 Checkpoint/restart API . . . . . . . 2.4.1 Initialization . . . . . . . . 2.4.2 Checkpoint parameters . . . 2.4.3 Checkpoint calls and control 2.4.4 Callback . . . . . . . . . . 2.4.5 Critical section . . . . . . . 3 Node-level VO support 11 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . .
IST-033576 A.1.5 Compiling BLCR . . . . . . . . . . . . . . . . A.1.6 Testing your build (optional, but recommended) A.1.7 Installing BLCR . . . . . . . . . . . . . . . . A.1.8 Loading the Kernel Modules . . . . . . . . . . A.1.9 Updating ld.so.cache . . . . . . . . . . . . . . A.1.10 Configuring Users’ environments . . . . . . . A.2 Uninstalling BLCR . . . . . . . . . . . . . . . . . . . A.3 Making RPMs from the BLCR sources . . . . . . . . . A.3.1 Building binary RPMs from the source tarball . A.3.
D2.1.4 1 IST-033576 Introduction The current prototype of Linux-XOS consists of two major components: the kernel checkpoint/restart mechanisms and the node-level VO-support mechanisms. Checkpointing in Linux-XOS is conceptually seperated in three layers: grid checkpointer, system checkpointer and kernel checkpointer. This document mainly describes the lower layer, the kernel checkpointer, as functionality for the system and grid checkpointer.
IST-033576 2.1 D2.1.4 Prerequisites BLCR consists of three kernel modules, some user-level libraries, and several command-line executables. No kernel patching is required. BLCR has been engineered to work with a wide range of Linux kernels: • Many major vendor distributions of Linux. Those tested historically have include RedHat 7.2 through 9, SuSE 9.x and 10.0, CentOS 3.1, and Fedora Core 2 through 4. • Many “vanilla” Linux 2.4.x and 2.6.x kernels (from kernel.
D2.1.4 IST-033576 2.4 Checkpoint/restart API The API described in D2.1.3 will become available with the next release. Therefore, checkpoint/restart API available is the BLCR API. The main functions of this API are described below, see header file libcr.h for a complete description of this API. However, BLCR functionalities have already been enhanced with the possibility to save libraries and executable during the checkpoint.
IST-033576 D2.1.4 Once done, user can specify values for the following fields of this structure in order to tailor the checkpoint to his need : • cr_scope: CR_SCOPE_PROC to checkpoint a process, CR_SCOPE_TREE for a process tree, CR_SCOPE_PGRP for a process group (as defined by POSIX, which typically means a command pipeline launched by a shell) and CR_SCOPE_SESS for a session (which typically means a login shell and all its descendants or a batch job).
D2.1.4 IST-033576 • EPERM: One or more of the target processes are not checkpointable by the current user • EBADF : The destination (args->cr_fd) is not a valid. • EINVAL: Some field of the args argument is invalid. • ... Check errno-base.h for more details. As the cr_request_checkpoint system call is not a blocking call, a cr_poll_checkpoint is provided to allow user to check or block for completion.
IST-033576 D2.1.4 • EIO: Problem writing the checkpoint output. • ENOSPC: Out of space on filesystem while writing the checkpoint 2.4.4 Callback If the user wants some specific logic to be executed before the dump, BLCR provides a way to register user-level callback function triggered whenever a checkpoint is about to occur, and which continue when a restart is initiated.
D2.1.4 IST-033576 • flags: used to specify whether the callback should be executed in a thread context (CR_THREAD_CONTEXT) or in a signal context (CR_SIGNAL_CONTEXT). All thread-based callbacks are runned before signal-based callbacks 2.4.5 Critical section BLCR provides user-level code with “critical sections” in order to allow groups of instructions to be performed atomically with respect to checkpoints.
IST-033576 D2.1.4 .tar.gz, which can be downloaded from https://gforge.inria.fr/frs/download.php/3648/xos-nss-pam-0.04.tar.gz. 3.2.1 Prerequisites The following Linux package must be present in order to install the XtreemOS NSS/PAM modules: Package automake autoconf libtool doxygen Minimal version >= 1.9 >= 2.59 >= 1.5.6 libc6-dev libpam libdb headers >= 4.3 keyutils >= 1.1 libkeyutils-dev libssl-dev Current version 1.10 2.61 1.5.22 1.5.1-1 2.3.6 0.79-4 4.4.20 1.2-3 1.2-3 0.9.
D2.1.4 IST-033576 yjegou: tar xzvf xtreemos-nss-pam-0.04.tar.gz yjegou: ls -F xtreemos-nss-pam-0.04/ xtreemos-nss-pam-0.04.tar.gz yjegou: cd xtreemos-nss-pam-0.04/ yjegou: ./configure ... yjegou: make ... yjegou: sudo make install ... The install step stores NSS modules in /lib/libnss_xos.so, PAM modules in /usr/local/lib/pam_xos.so and some configuration files in /etc/xos. The binaries are installed in directory /usr/local/bin. Compilation and installation of test programs.
IST-033576 D2.1.4 reference to the pam_xos module can be added to the /etc/pam.d/su configuration file of su in order to add xos authentication capability to the su command: root: cat /etc/pam.d/su #%PAM-1.0 ... auth sufficient ... root: /usr/local/lib/pam_xos.so For this validation, only the two experimental codes delivered with this prototype, pam_app_conv and ssh-xos, will be configured to use XtreemOS. The behaviour of this first prototype can also be configured in the /etc/xos directory: pam_xos.
D2.1.4 IST-033576 ----You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter ’.’, the field will be left blank.
IST-033576 D2.1.4 4E:7B:C5:84:07:26:B3:2E:51:06:5C:2A:CE:DC:53:11: FB:20:A1:62 X509v3 Authority Key Identifier: keyid:4E:7B:C5:84:07:26:B3:2E:51:06:5C:2A:CE:DC: 53:11:FB:20:A1:62 DirName:/C=EU/ST=France/L=Rennes/O=INRIA/OU=IRIS A/CN=XtreemOS-test-CA/emailAddress=Yvon.Jegou@irisa.fr serial:96:87:93:B3:BF:7C:49:49 X509v3 Basic Constraints: CA:TRUE Signature Algorithm: sha1WithRSAEncryption 2d:c6:ad:97:1d:22:c0:8e:3c:f1:97:6a:a4:7a:64:34:f4:0e: ...
D2.1.4 3.3.2 IST-033576 User certificate generation Generate a user certificate request: yjegou: openssl req -new -days 365 -keyout security/private/XtreemOS-Yvon-key.pem -out security/XtreemOS-Yvon-req.pem Generating a 1024 bit RSA private key .....................................++++++ .....++++++ writing new private key to ’security/private/XtreemOS-Yvon-key.
IST-033576 D2.1.4 -CAkey security/private/XtreemOS-ca.key -CAcreateserial -days 365 Signature ok subject=/C=FR/ST=Bretagne/L=Rennes/O=INRIA/OU=IRISA/CN=Yvon Jego u/emailAddress=Yvon.Jegou-at-irisa.fr Getting CA Private Key Enter pass phrase for security/XtreemOS-ca/private/XtreemOS-Yvon .
D2.1.
IST-033576 D2.1.4 Signature ok subject=/C=FR/ST=Bretagne/L=Rennes/O=INRIA/OU=IRISA/CN=Yvon Jego u/emailAddress=Yvon.Jegou-at-irisa.fr/CN=28216 Getting CA Private Key Enter pass phrase for security/XtreemOS-Yvon/Yvon-key.pem: success in creating proxy cmd --> 3 ---- 3. Set attributes in proxy ----Input proxy: .xos/firstproxy.pem Input attributes: /VO=xtreemos/ROLE=admin Generating RSA private key, 512 bit long modulus ..........++++++++++++ .
D2.1.4 IST-033576 pam_xos.conf configuration file: The certificate verification chain for pam_xos.so PAM plugin is configured in file /etc/xos/pam_xos.conf, item VOCAPublicKeyfile. By default, this item defines /etc/xos/certificates/ as the CA public key directory. root: cat /etc/xos/pam_xos.conf ... #VOACConf /etc/xos/mapdata/quota.conf VOCAPublicKeyfile /etc/xos/certificates/ #NodePrivateKeyfile /tmp/userkey.
IST-033576 D2.1.4 pam_xos.c:119: PAM: configure file is: /etc/xos/pam_xos.conf =============== PAM configure data =============== work mode: Use external AMS -- External AMS server: localhost -- External AMS port: 8000 VO CA dir: /etc/xos/certificates/ ================================================== pam_xos.c:594: PAM:Get information,dn->[/C=FR/ST=Bretagne/L=Renn es/O=INRIA/OU=IRISA/CN=Yvon Jegou/emailAddress=Yvon.Jegou-at-iri sa.
D2.1.4 IST-033576 drwxrwxrwt 4 root root 4096 Oct 31 09:31 . drwxr-xr-x 20 root root 4096 Oct 31 08:24 .. drwxrwxrwt 2 root root 4096 Oct 31 08:24 .ICE-unix drwxrwxrwt 2 root root 4096 Oct 31 08:24 .X11-unix -rw-r--r-- 1 /C=FR/ST=Bretagne/L=Rennes/O=INRIA/OU=IRISA/CN=Yvo n Jegou/emailAddress=Yvon.Jegou-at-irisa.fr/CN=28216/CN=10098738 018704381327 xos_admin 0 Oct 31 09:31 afile XXX: env TERM=xterm SHELL=/bin/bash SSH_CLIENT=131.254.13.
IST-033576 D2.1.4 pam_xos.c:119: PAM: configure file is: /etc/xos/pam_xos.conf =============== PAM configure data =============== work mode: Use external AMS -- External AMS server: localhost -- External AMS port: 8000 VO CA dir: /home/yjegou/security/XtreemOS-ca ================================================== pam_xos.c:784: PAM:Get attrs->[/VO=xtreemos/ROLE=admin] pam_xos.c:791: PAM:role = [admin] pam_xos.c:809: PAM:finish policy clear ... root: 3.
D2.1.4 IST-033576 root: ./xtreemos-nss-pam-0.04/src/test/pam_app_conv -pem ~yjegou/.xos/firstproxy.pem vo = [xtreemos], role = [admin] pam_xos.c:239: PAM:pam_sm_authenticate: uid = 0, user = [root] ERROR: Verifying certificate chain: unable to get local issuer certificate xos_keyring_revoke: No content in keyring pam_xos.c:488: PAM: ...
IST-033576 D2.1.4 pam_xos.c:613: PAM:fail in mapping connect Oops: Per mission denied If the pam_app_conv test program produces this error, the amsd service is not running (as root) or is mis-configured. root: ./xtreemos-nss-pam-0.04/src/test/pam_app_conv -pem ~yjegou/.xos/firstproxy.pem vo = [xtreemos], role = [admin] pam_xos.c:239: PAM:pam_sm_authenticate: uid = 0, user = [root] Verifying certificates ... OK pam_xos.c:568: PAM:pam_sm_acct_mgmt: uid = 0, user = [root] pam_xos.
D2.1.4 3.6.2 IST-033576 Configuration PAM: XOS-ssh is a PAM-aware operating system service. This service reads its PAM configuration from file /etc/pam.d/sshd-xos. This file must contain the following items: root: cat /etc/pam.d/sshd-xos ... auth sufficient /usr/local/lib/pam_xos.so account sufficient /usr/local/lib/pam_xos.so session sufficient /usr/local/lib/pam_xos.so ... root: Ssh-xos: The default configuration file for ssh-xos is /usr/local/etc/ssh_config-xos.
IST-033576 3.6.3 D2.1.4 Running ssh-xos Start sshd-xos service: root: /usr/local/sbin/sshd-xos -d -f /usr/local/etc/sshd_configxos debug1: sshd version XOS-OpenSSH_4.
D2.1.4 IST-033576 xos_buffer_to_keyring: xos_buffer_to_keyring:keyctl_updatepam_xo s.c:510: PAM:Can not store proxy in keyring ! debug1: PAM: establishing credentials xos_buffer_to_keyring: xos_buffer_to_keyring:keyctl_updatepam_xo s.c:537: PAM:Can not store proxy in keyring ! debug1: permanently_set_uid: 55285/50376 debug1: PAM: reinitializing credentials pam_xos.c:502: PAM:pam_sm_setcred: Get proxy [4382] : /n-----BEG IN CERTIFICATE----MIIC7jCCAlegAwIBAgIJAIwl7Lj8m1WPMA0GCSqGSIb3DQEBBQUAMIGdMQswCQYD ...
IST-033576 root root root yjegou 55285 55285 root D2.1.4 25731 25748 25906 25907 25914 25924 25925 25730 25731 14038 13968 25906 25914 25748 0 0 0 0 0 0 0 17:05 17:05 17:29 17:29 17:29 17:35 17:35 pts/2 pts/2 pts/7 pts/8 pts/5 pts/5 pts/2 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 bash ./xos_amsd -d sshd-xos: xosuse ssh-xos paraski3 -bash ps -aef ./xos_amsd -d /C=FR/ST=Bretagne/L=Rennes/O=INRIA/OU=IRISA/CN=Yvon Jegou/emailA ddress=Yvon.Jegou-at-irisa.
D2.1.4 IST-033576 VOs. It is a proof-of-concept of extending standard Linux to treat VO users as local accounts in a transparent and application-independent way. The current prototype has limited support for local policies of access control and resource usage constraints for VO users. In next steps, the enforcement of local policies against VO users will be the focus.
IST-033576 A D2.1.4 Berkeley Lab Checkpoint/Restart (BLCR) Administrator’s Guide This guide (extract from the doc directory of the BLCR tarball) describes how to install, configure, and maintain Berkeley Lab Checkpoint/Restart (BLCR) for Linux. A.1 Installing/Configuring BLCR To build checkpoint/restart, you need the following files: • The source code for the configured kernel you are building against. • linux/version.h (a generated file from the kernel sources). • either the System.
D2.1.4 IST-033576 We strongly recommend that you configure and build BLCR in a directory other than the one containing the BLCR source code (use of some options to configure actually require this). In the example above the build is conducted in a subdirectory, named builddir, of the source directory. Any writable location is fine, but you will need to invoke configure by the correct path in place of ../configure used in the example. Check the FAQ if you run into issues building BLCR on your system. A.1.
IST-033576 A.1.4 D2.1.4 Building 32-bit application support on a 64-bit platform BLCR’s build logic is capable of building both 64-bit and 32-bit libraries at the same time on most 64-bit platforms it supports. However, because this feature is new and does not work well with certain setups, it is disabled by default.
D2.1.4 IST-033576 tests that say "restart/timeout" then you should first try increasing the timeout as follows (assuming the kernel modules have already been loaded): % make c h e c k CRUT_TIMEOUT=120 The CRUT_TIMEOUT is a value in seconds, with a default of 60 (CRUT is an acronym for Checkpoint/Restart Unit Test). Tests marked SKIP are neither a PASS, nor a FAIL - instead they indicate a test that was not actually run. So don’t be alarmed if you see one or more tests marked SKIP.
IST-033576 D2.1.4 You may wish to set up your system to load these modules by default at boot time. The exact mechanism for doing so differs between Linux distributions, and thus requires an experienced system administrator. However, a template init script is provided as etc/blcr.rc in the BLCR source directory. A.1.9 Updating ld.so.cache Nearly all Linux distributions use a caching mechanism for resolving dynamic library dependencies.
D2.1.4 IST-033576 % s e t e n v PATH $ {PATH } : PREFIX / b i n % s e t e n v MANPATH $ {MANPATH} : PREFIX / man % s e t e n v LD_LIBRARY_PATH $ {LD_LIBRARY_PATH } : PREFIX / l i b A.2 Uninstalling BLCR If you preserve the BLCR build tree, then there is a standard uninstall make target available to remove the files copied by the install target. A.3 Making RPMs from the BLCR sources An alternate way to install BLCR is to build a binary RPM for your system, which you can then install.
IST-033576 D2.1.4 % r p m b u i l d −−r e b u i l d b l c r −X . Y . Z−N . s r c . rpm −− t a r g e t ARCH replacing blcr-X.Y.Z-N.src.rpm with the correct filename, and ARCH with a specific target CPU. If you don’t know your target, try uname -p to determine it. If you don’t specify a –target, the default will depend on the version of rpmbuild and may be i386 (which will be rejected). See the documentation for rpmbuild for more information on building binary RPMs from source RPMs.
D2.1.4 B IST-033576 Berkeley Lab Checkpoint/Restart (BLCR) User’s Guide B.1 About Berkeley Lab Checkpoint/Restart Checkpoint/Restart allows you to save one or more processes to a file and later restart them from that file. There are three main uses for this: • Scheduling: Checkpointing a program allows a program to be safely stopped at any point in its execution, so that some other program can run in its place. The original program can then be run again later.
IST-033576 D2.1.4 Unfortunately BLCR has not yet been integrated with many batch systems. Support for serial jobs is available through SGE. See this report for more information. As with MPI implementations, efforts are under way to integrate BLCR with additional batch systems, so check your batch system’s documentation for the latest info. The rest of this document assumes that your batch scheduler does not have built-in support for BLCR.
D2.1.4 IST-033576 % c r \ _ c h e c k p o i n t −−h e l p If cr_checkpoint cannot be found, you need to modify your PATH to include the directory where cr_checkpoint lives. You will probably also want to modify your LD_LIBRARY_PATH variable to contain the directory where libcr.so lives, and add the BLCR man directory to your MANPATH. B.3.
IST-033576 D2.1.4 The above examples set the PATH, MANPATH and LD_LIBRARY_PATH variables in your current shell only. It is strongly recommended that you make these settings permanent, to make these settings affect future sessions or windows. To do this, you must add the example commands to your shell’s start up files. For a single user of BLCR, you should add the appropriate set of commands to the shell startup files in your home directory (.bashrc for bash, .profile for other bournetype shells, or .
D2.1.4 IST-033576 • Others - this list is not exhaustive. If you have questions about specific resources, see the section "For more information" for contact information. B.4.2 Making an application checkpointable To be checkpointed successfully with BLCR, an application must contain some library code that BLCR provides. There are several ways of ensuring this: 1.
IST-033576 D2.1.4 % env LD_PRELOAD=PREFIX / l i b / l i b c r . s o . 0 : l i b p t h r e a d . so . 0 y o u r _ e x e c u t a b l e [ arguments ] This is essentially how cr_run works. If you your application does not link in BLCR’s library via one of the mechanisms listed above, then any attempt to checkpoint it will fail gracefully NEW: This behavior is a change BLCR releases prior to 0.5.0 in which this situation would cause the program to die unless you handled BLCR’s real-time signal explicitly. B.4.
D2.1.4 IST-033576 or session leader. When in doubt, try using the ’-j’ option to ’ps’ to show PGID and SID columns. The –tree flag to cr_checkpoint requests a checkpoint of the process with the given pid, and all its descendants (excluding those who’s parent has exited and thus become children of the init process). This is the same as the grouping shown by the output of the pstree command. NEW: The behavior in 0.6.0 is a change from previous BLCR releases in which the –pid option (below) was the default.
IST-033576 D2.1.4 • Files must exist at their original paths and have permissions that permit them to be opened for the original access modes. It is permissible to use a copy of the original file (for instance when performing migration). • Files open for reading should not be modified relative to their contents when the checkpoint was taken in any way observable by the application.
D2.1.4 B.5 IST-033576 Checkpointing/restarting an MPI application The best source of information on dealing with any BLCR-aware MPI implementation is the documentation provided with the MPI. However, here are some hints that may be helpful. B.5.1 Checkpoint/restart with LAM/MPI • See the the LAM/MPI documentation for the most detailed info on using LAM/MPI with BLCR. • When building your own LAM/MPI, do NOT configure LAM to debug mode, i.e. do not pass –with-debug to LAM’s configure script.
IST-033576 D2.1.4 B.6 For more information For more information on Berkeley Lab Checkpoint/Restart for Linux, visit the project home page: http://ftg.lbl.gov/checkpoint To report bugs (or look for bug fixes prior to reporting new ones), visit http://mantis.lbl.
D2.1.4 IST-033576 References [1] XtreemOS Deliverables - D2.1.2: Design and Implementation of Node-level VO Support. [2] XtreemOS Deliverables - D2.1.3: Design and Implementation of basic checkpoint/restart mechanisms in Linux. [3] Jason Duell. The design and implementation of berkeley lab’s linux checkpoint/restart.