Chapter 3. Installation

3.1. Introduction

This section describes how to download, compile and install the GRASP Open software from the user's perspective. Basic knowledge of the terminal and some tools such as GIT are necessary to complete this process.

3.2. Hardware requirements

The system has been tested on modern Intel architectures only (PC compatible, Macintoshes). It should work on any 32-bit and 64-bit platform. A minimum of 2 GB of RAM is recommended. GRASP is a very hardware demanding software because math calculations are very time-consuming. In a multicore scenarios GRASP is able to parallelize the entire process using MPI technology. Additionally, GRASP offers a GPGPU module (as extension, not installed by default) that allows to parallelize a single segment retrieval using graphics cards.

3.3. Operating Systems

The code has been widely tested under Linux machines and MacOSX™ systems. The installation process in old operating systems is usually more complex so we suggest to always use the lastest version of operating system. Ubuntu systems have shown the easiest installation process so we recommend it for regular users.

For windows users some test under Microsoft Windows™ has been also performed successfully using Cygwin, but we do not offer official support for this solution. Some manipulations are needed depending on the specific version of Windows and Cygwin, so even if a solution is possible, it is not straightforward and we do not recommend it. There is one exception: the last version of Microsoft Windows 10™ offers the feature Windows Subsystem for Linux which allows the users to set up an Ubuntu environment over Windows. This unestable but very promissing feature can really help to install GRASP on Windows. You can start with this solution following this article of MSDN. Once you have your Ubuntu environment configured, you can follow this guide as a Linux user.

3.4. Access to GRASP Open repository

To access the code, the users have to register their account on GRASP Open web page. There are different ways to download the code (direct download link, clone over HTTP...) but we recommend to clone the code over GIT protocol. This method has the benefit of being manageable by the grasp-manager (see Section 3.7.1, “GRASP Manager”) which is the easiest way to keep the code up-to-date and manage GRASP extensions. The following steps show the process of getting access to the repository and setting up your access via GIT protocol.

  1. Go to GRASP-Open web page code section. And click in the "GITLAB REPOSITORY" bottom.
  2. Fill all the fields of the registration form (last section of that page), accept the conditions and press SIGNUP button.
  3. If the registration was successful, you will get a confirmation message. Click on the "Initialize password" button.
  4. The GitLab system, which is the system used to manage the code, will ask you for your email to reset the password (initialize it). Introduce your email and click on the "Reset Password" button.
  5. You'll receive an email for resetting the password. Follow the link in the email and set up your new password.
  6. The next window will show the main page of the GitLab system. You can sign in using your email and your password. Remember that you can access that web page at anytime, using the button "GITLAB REPOSITORY" in GRASP-Open web page code section.
  7. On the main page of the GitLab system you can see different repositories you can access. Go to the GRASP repository or use this link
  8. There you can explore many things: you can see the code, see the changes, open an issue to get in touch with the developer team ... To download the code, there are three alternatives: a) direct download, b) cloning repository over http or c) cloning repository over git protocol. The last solution is recommended so we'll continue explaining this process.
  9. Set up your ssh-key in GitLab. It allows you to access the code via GIT protocol whitout a need of typing passwords. To set up a ssh-key you have to go to you-profile>edit>ssh-keys section or using this link.
  10. Follow this guide to create a ssh-key if you don't have one. If you already have a ssh-key you can follow the same guide but skip the first steps of a key creation, just copy it to the clipboard.
  11. Paste your ssh-key in the GitLab system and click the button "Add key"
  12. Now your system is properly configured and you can download the code over the git protocol. The section Section 3.5.2, “Basic installation of GRASP explains how to download the code, compile and install it.

3.5. Building and installing GRASP

The GRASP software makes a heavy use of a number of libraries for data preparation and numerical computations. As for the GRASP software itself, it relies on the following libraries FOSS or belong to the public domain.

We are very focused on keeping the whole system free and depending only on non-commercial libraries. That does not mean that non-open source software can't be linked to GRASP (certainly closed-source, or even open-source, although restricted [2] solutions are sometimes better than the free ones for certain purposes), but the system should always be able to run only with free, open-source alternatives.

3.5.1. Dependencies

The following list shows the GRASP core dependencies. Some extensions can require extra dependencies, in that case, please follow the documentation of the extension to know the installation process.

  • a C compiler (recommended gcc).
  • a Fortran compiler (known to work with gfortran and ifort).
  • a make command (provided on any POSIX system).
  • the cmake building software.
  • One of the four numerical packages:

  • a BLAS library (Netlib BLAS, ATLAS, GotoBLAS). Actually, BLAS is not used directly by GRASP, but some numerical packages (SuperLU, MUMPS), which GRASP relies on, are built on the BLAS. BLAS will be necessary only if you build GRASP with these packages. Currently, Netlib BLAS is the default BLAS library for the GRASP but this may change over time. The GRASP code has currently been tested with the Netlib, BLAS and ATLAS.
  • The LAPACK library. As for the ATLAS, LAPACK is only necessary with the SuperLU and MUMPS packages, not with ViennaCL. Please note that ATLAS ships with a partial LAPACK implementation for its own purposes, but that is not sufficient for the numerical packages on which GRASP relies. You must install the full LAPACK package.
  • ScaLAPACK (a requirement for the SuperLU and MUMPS numerical packages only).
  • ParMETIS (a requirement for the MUMPS numerical package only).
  • OpenMPI (optional). This package is needed to work with the MUMPS. The GRASP software can also be compiled using MPI by adding it the capability to process many segments at the same time.
  • The Gnome Library GLib (mostly used by the configuration manager).
  • The LibYAML library (YAML is the chosen format for GRASP configuration).
  • The C Unit Testing Framework: CUnit.

These dependencies can be installed in an Ubuntu system with the following command:

        

# All deps except superlu:
sudo apt-get install build-essential cmake git gfortran libyaml-dev libglib2.0-dev libcunit1-dev libsuperlu-dev
                

3.5.2. Basic installation of GRASP

Your system should be ready now for the installation. If it is not the case, please refer to the previous section. Also, it is assumed that the user has access to the git repository, otherwise please check Section 3.4, “Access to GRASP Open repository”.

The steps to download and install GRASP are the same for all platforms (Windows/Cygwin, MacOSX, Linux). Depending on your system, you may or may not have the sudo command. It is used for running a command with administrative rights. In that case, you can try without sudo (e.g. on Cygwin), or use the su instead for logging as administrator. You can also perform a custom installation (see the next section) so you don't need to be administrator. If none of this makes sense for you, ask your local Unix guru.

        

$ git clone git@code.grasp-open.com:open/grasp.git 
$ cd grasp
$ # you should now be in the master branch of the project
$ # (developers of the project may need to checkout the dev branch)
$ make # build the project using the default build settings
$ sudo make install # install grasp. Administrative privileges are needed.
                

3.5.3. Advanced compilation

GRASP uses cmake system to compile the code. You can compile the code using CMAKE following these steps:

        

$ # Place a terminal in GRASP root folder
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_BUILD_TYPE=Release -DADDITIONAL_DEPENDENCIES_PATH=/usr/local/grasp-deps -DCONSTANTS_SET=generic
$ make -j12
$ sudo make install
$ grasp # test the command
                

You can use a different compilation configuration. All configuration options are defined in Section 3.5.3.2, “Custom installation using cmake”

For users that are not familiar with cmake there is a Makefile which wraps cmake system and is placed in root folder. This Makefile simplifies the use of cmake via make script. Internally, the Makefile creates cmake structure and call it. Thanks to this operation, you can compile using this system like it is defined in Section 3.5.2, “Basic installation of GRASP. Addionally, this Makefile also allows you to use some extra configuration parameters that are explained in Section 3.5.3.1, “Custom installation using make”.

Finally, the last way to compile the code is via grasp-manager. Grasp-manager is described in the next section: Section 3.7.1, “GRASP Manager”

.

3.5.3.1. Custom installation using make

If you have compiled the code following the rules explained in Section 3.5.2, “Basic installation of GRASP, you have compiled the code with default options. The options and their possible values are listed below (the default value is written in italic):

  • CONSTANTS_SET: generic or see Section 3.5.3.3, “Constants sets”
  • BUILD: Release or Debug, RelWithDebInfo or Fast
  • MPI: off or on
  • DEBUG_MPI: off or on
  • F90: gfortran or ifort
  • PREFIX: /usr/local or other valid path where dependencies are available
  • BUILD_DIR: build or other name but then build is ignored by git
  • CC: cc or another valid c compiler
  • CCX: c++ or another valid c++ compiler

For example:

        

$ make CONSTANTS_SET=polder MPI=on
                    

3.5.3.2. Custom installation using cmake

By default, the resources and dependencies will be installed in the following directories:

/usr/local/share/grasp (resources that are internal databases or files used by GRASP)
/usr/local/grasp-deps (for general-purpose, utility libraries)

If one does not wish (or may not be able) to install GRASP under default system directories, it is possible, and very easy to change these paths of installation with the PREFIX variable.

Please find below the way to install the project under your HOME directory instead of /usr/local (now you don't need administrative rights anymore). In this case, we will use cmake compilation system instead of the Makefile placed in the root folder, which wraps it.

        

$ cd ~/grasp/dependencies
$ sudo make PREFIX=$HOME/local install
$ #The command above builds and installs the third-parties dependencies under /home/your_name/local,
$ #instead of /usr/local
$ cd ..
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_BUILD_TYPE=Release -DADDITIONAL_DEPENDENCIES_PATH=$HOME/local -DCONSTANTS_SET=generic
$ #The command above generates a Makefile with custom parameters
$ make -j12 # build the project
$ sudo make install # install grasp under $HOME/local/bin
$ $HOME/local/bin/grasp
$ # test the command (of course, it is then recommended to add $HOME/local/bin to your PATH)
                    

Another possible customization is to change the numerical solver for GRASP. By default, it is set to SUPERLU. If you wish to try another solver [3], you may use:

cmake SPARSE_SOLVER= your_chosen_solver

Table 3.1. SPARSE_SOLVER's valid values

SUPERLU (default)
SUPERLU_MT (not supported yet)
VIENNA_CL
MUMPS

You may also want to link the project to the BLAS implementation of your choice. This is possible with:

cmake BLAS= your_chosen_blas_library

Table 3.2. BLAS valid values

netlib-blas (default)
atlas

The CONSTANTS_SET setting can be used for performing memory optimizations with a specific set of data, e.g. cmake CONSTANTS_SET=polder. If one is not sure, one may stick to the default. See Section 3.5.3.3, “Constants sets”

Table 3.3. Main CONSTANTS_SET values (installed by default)

generic (default)
valgrind
(you can install external constants set and use them)

The BUILD setting can be used to switch between debug (non optimized, with debug information), dev (partially optimized) and prod (optimized) modes at build time, e.g. cmake BUILD=prod. The default is dev.

Table 3.4. BUILD valid values

Debug
Release (default)
RelWithDebInfo
Fast

Of course, there is no limitation to the number of settings that the cmake command can accept:

cmake PREFIX=$HOME/local BUILD=prod SPARSE_SOLVER=SUPERLU CONSTANTS_SET=generic

3.5.3.3. Constants sets

GRASP is a highly memory- and time-consuming software, which is strongly optimized. Usually in software development, time can be optimized using more memory and memory can be saved using more calculation time. In software development, there should be a balance between the time consumption and the memory cost. GRASP has found this balance using many static arrays to optimize time performance and defining the size of these arrays like constant values. This method permits to choose during compilation time, the size of these arrays, allowing the user to optimize the software, depending on the use. For example, GRASP will take more memory if it is compiled to use a maximum of eight wavelengths instead of four. We have created some standard constants sets for the most common uses and in general you can use "generic" CONSTANTS_SET as one valid to test all instruments. If it is not valid you can check whether your application fits with another existing constants set or create your own. Constants sets are managed by grasp-manager as an extension of GRASP. It means that you can install an external definition of the constants set depending on your necessities or create your own. Constants set are placed in src/retrieval/constants_set/{CONSTANTS_SET_NAME} and consist of four files:

  • mod_par_DLS.inc
  • mod_par_OS.inc
  • mod_par_DLS_bin.inc
  • mod_par_inv.inc

Where {CONSTANTS_SET_NAME} is the name of the constants set, which also can be defined by the user, for code compilation.

3.6. Running the code

Once the code is installed, you can test it right away. In the following examples, it is assumed that the user has installed GRASP as a wide system executable. Otherwise, the GRASP binary is placed in ./build/bin/grasp. When GRASP is executed without arguments, it prints some general information about the version and how it was compiled:

    

$ grasp

GRASP core version: v0.7.0 (commit: c0bd56a ; branch_name: HEAD)
Compiled on 2016-06-02 10:21:03 +0200 commit of 2016-05-24 18:17:11 +0200
With C compiler: Apple LLVM version 7.3.0 (clang-703.0.31)
With FORTRAN compiler: GNU Fortran (Homebrew gcc 5.3.0 --with-all-languages) 5.3.0
Using generic constant set and build type Release
Maximum segment size: nx=2 ; ny=2 ; nt=30
Input drivers loaded: sdata
Input transformers loaded: none
Output segment functions loaded: ascii classic classic_plot none
Output tile functions loaded: ascii none
Output current functions loaded: none
Path to resources: /usr/local/share/grasp/
Sparse solver used: SuperLU
Build System: Darwin-15.5.0
Executable path: undefined absolute path (./build/bin/grasp_app)

usage: grasp [OPTIONS] {settings_file.yml}|help

OPTIONS:
-V       use valgrind
            

In the examples folder there are some examples that the user can apply to verify that the whole system is working properly. To run an example you have to call GRASP followed by the settings file as the first argument:

    

$grasp examples/sunphotometer/settings_example_sunphotometer_inversion.yml
Config file read successfully 1
The tile is divided in segments with 1 rows, 1 cols and 1 times. 1 inversions will be performed (sequential version) 2
Retrieval #0 (1/1): 100.00%: 1 pixel will be processed 3

... 4

826.69409  1:   0.76332E+00   84.82804 %    2:   0.47585E+00    119.16735 %    pixel # 1      Residual using INITIAL GUESS
277.77368  1:   0.29931E+00   30.63890 %    2:   0.33364E+00    30.73591 %   pixel # 1    Residual after iteration # 1
22.29646  1:   0.71556E-01    5.62210 %    2:   0.21300E-01     2.00627 %    pixel # 1    Residual after iteration # 2
8.99939  1:   0.19544E-01    1.39876 %    2:   0.86748E-02      1.84197 %    pixel # 1    Residual after iteration # 3
1.26664  1:   0.34922E-02    1.21845 %    2:   0.25801E-03      0.03841 %    pixel # 1    Residual after iteration # 4
0.91952  1:   0.27402E-02    0.83636 %    2:   0.34244E-03      0.05860 %    pixel # 1    Residual after iteration # 5
0.49750  1:   0.27118E-02    0.39311 %    2:   0.28940E-03      0.09854 %    pixel # 1    Residual after iteration # 6
0.49593  1:   0.22867E-02    0.25629 %    2:   0.41266E-03      0.10136 %    pixel # 1    Residual after iteration # 7
0.37954  1:   0.83430E-03    0.12501 %    2:   0.34994E-03      0.03372 %    pixel # 1    Residual after iteration # 8
0.19446  1:   0.67640E-03    0.09489 %    2:   0.16508E-03      0.04304 %    pixel # 1    Residual after iteration # 9
0.11306  1:   0.59844E-03    0.07205 %    2:   0.84066E-04      0.02038 %    pixel # 1    Residual after iteration # 10
0.11299  1:   0.58013E-03    0.07008 %    2:   0.85618E-04      0.01960 %    pixel # 1    Residual after iteration # 11
Retrieval #0 (1/1): 100.00%: 1 pixels processed in 15.478408 seconds (cpu time: 15.433357).
Average per pixel: 15.478408 (cpu time: 15.433357)
Retrieval #0 (1/1): 100.00%: finished

... 5

Size Distribution dV/dlnr (normalized to 1) for 1 - fraction
0.50000E-01   0.18937E-03
0.65604E-01   0.25878E-01
0.86077E-01   0.31692E+00
0.11294E+00   0.46703E+00
0.14818E+00   0.49857E+00
0.19443E+00   0.31519E+00
0.25510E+00   0.13060E+00
0.33472E+00   0.37423E-01
0.43917E+00   0.15058E-01
0.57623E+00   0.21979E-01
0.75605E+00   0.60639E-01
0.99200E+00   0.12603E+00
0.13016E+01   0.20535E+00
0.17078E+01   0.31518E+00
0.22407E+01   0.32588E+00
0.29400E+01   0.32922E+00
0.38575E+01   0.22504E+00
0.50613E+01   0.15226E+00
0.66407E+01   0.84118E-01
0.87131E+01   0.25702E-01
0.11432E+02   0.33366E-02
0.15000E+02   0.15965E-03

... 6

Total Time: 1 pixels processed in 15.573089 seconds (cpu time: 15.515442).
Average per pixel: 15.573089 (cpu time: 15.515442)
Algorithm Time: 1 pixels processed in 15.478408 seconds (cpu time: 15.433357).
Average per pixel: 15.478408 (cpu time: 15.433357)
Control Unit Time: 1 pixels processed in 0.094681 seconds (cpu time: 0.082085).
Average per pixel: 0.094681 (cpu time: 0.139732)
        

1

First step of execution is to parse settings file and validate it. If everything is OK this line will be printed, otherwise a list of errors will be produced. Please, pay attention to error messages because they should help you to understand what is going on and how to resolve the problem.

2

This line informs you how the data are going to be organized for retrieval. Data are organized as segments that will run inside the retrieval algorithm. Control unit will organize these segments into a tile as a bigger group of pixels.

3

The first retrieval is launched and the retrieval algorithm starts to process the first segment (group of pixels)

4

The retrieval can be in a verbose mode or not. If it is in a verbose mode, a detailed information on the process will be printed. The most important information is in the next lines. The user can follow the fitting process and see how the errors decrease iteration by iteration.

5

Finally, the results are processed. This example prints results on the screen. You can dump them to a file in different formats such as CSV, HDF ... Output functions are extensions that you can optionally install. In the settings file, it is defined which functions are used and how (set up).

6

The process is finished with a small summary about how many data have been processed and how long it took.

It is also possible to add any number of arguments in the form of setting=value. They provide a quick and easy way to override the default settings in the configuration file, for experimenting without editing this file.

Table 3.5. Some common settings

NameTypeDefaultDescription
help boolean [a] falseWhen set to true, displays an exhaustive list of settings with their significance.
input.debug.print_clean_segment booleanfalseWhen set to true, prints a segment that has just been cleaned from its non-significant data. (see also print_raw_segment)
input.debug.print_raw_segment booleanfalseWhen set to true, prints a segment that has just been loaded from a driver (with possible non-significant data). (see also print_clean_segment)
input.sdata.dump booleanfalseWhen set to true, displays the input measurements in the form of a SDATA stream. Mostly useful for the maintainers of the scientific subsystem.
input.sdata_driver.debug booleanfalseWhen set to true, displays the actions of the sdata driver when a SDATA file is being read. This can be used for validating a new SDATA file (the contents of this file will be displayed in a readable form), and less commonly for debugging the SDATA driver.
retrieval.debug.print_segment_information booleanfalseWhen set to true, prints a segment with data that are actually passed to the retrieval library (unless there is a bug in the C/Fortran interface, the data should be the same as those from input.debug.print_clean_segment).
retrieval.debug.verbose booleantrueWhen set to true, displays debugging information relative to the retrieval subsystem. Mostly useful for the maintainers of the scientific subsystem.

[a] Boolean variables (also known as logical in the FORTRAN community) can take the values true or false (that can also be abbreviated as t and f)


3.6.1. Usage of GRASP: The configuration file

The default behaviour of the system is defined in a configuration file, whose settings can be overridden by command line arguments, as described in the previous section. A configuration manager (also known as the settings module) is responsible for loading the configuration file and for taking care of the overriding mechanism for command line arguments, if necessary. It centralizes all the information needed for a processing and in that matter drives the actual behaviour of the framework.

When there is more than a few parameters to change, or when one wishes to perform more persistent corrections, it is easier to edit the configuration file than to set parameters in the command line.

The chosen format for new GRASP configuration files is YAML, a standard format perfectly adapted for complex configurations such as the one needed by the GRASP project. It provides support for simple values as well as for complex data structures, while maintaining a high level of readability. YAML format is based on fixed indentation (the spaces before an element define the level where it applies). The command line interface of GRASP proposes an easy system to overwrite the main settings file with a "dot syntax", where each level of indentation is replaced by a dot. For example, in GRASP, it is equivalent to be called with the argument input.driver=sdata or to be defined in the settings file, as follow:

input:
    driver: sdata
                

GRASP settings system allows to import external files into one. Note that the standard YAML format doesn't support file inclusion, but GRASP configuration files support this feature through the import keyword that expects a list of files to be included. It is, therefore, possible to split large configuration files into smaller, easier to maintain independent files (this is especially important since several people will have to maintain different sections of the configuration).

Figure 3.1. Excerpt of configuration file

import: [ ] 1

input: 2
    driver: sdata_driver
    filename: bin/SDATA_NEW.dat

retrieval:
    # General retrieval parameters 3
    general:
        minimization_convention: logarithm
        threshold_for_stopping: 1.0e-5
        number_layers: 50 4
        shift_for_applying_logarithm_to_negative_values: 0.2
        binning_method: logarithm
        maximun_iterations_of_Levenberg-Marquardt: 15
        stop_before_performing_retrieval: false
        internal_file_path: "../retrieval/internal_files/"
        external_file_path: "../../home/"
        reference_plane_for_polarization: meridian
        regime_of_measures_fitting: absolute_polarization_components
        linearization_threshold:  0.03
        IMQ:       2
        use_internal_initial_guess:     false     
        threshold_for_length_corrections:     -1.0e-2
        threshold_for_stopping_Q_iterations:  1.0e-2
        scale_for_finite_difference:          1.0e-3
        irradiance_corrected: false # or no or 0
        coeff_corr: 0.96
        regime_of_multipixel_constraints: 
            inversion_regime: multi_pixel_followed_by_single_pixel  
            time-scale:   100.01
            x-scale:   100.05
            y-scale:   100.05
        error_estimation: true   
        number_of_characteristics_retrieved: 7	
...     
    noises:
        noise[1]: 5
            standard_deviation: 0.0
            error_type:  absolute
            variation:  0.01
            measure_type[1]:
                type: I
                wavelength_involved: [ 1, 2, 3, 4, 5, 6 ] 6
        noise[2]:
            standard_deviation: 0.0
            error_type:  absolute
...
                

1

The GRASP configuration file supports the file inclusion, with the import statement. The included files must be given in a comma-separated list, this can remain empty.

2

The configuration is organized in sections. The structure is defined by the explicit indentation. The order of the sections and the elements in the sections doesn't matter, as long as the structure and indentation are respected.

3

Comments are supported: anything after a # sign is concidered as a comment.

4

Access to fields from the command line (for overriding a setting, say) is straightforward, using the common dotted notation: for instance, the fully qualified name of number_layers is retrieval.general.number_layers

5

Section and property tags can be indexed, with a simple bracket notation. A feature that is not supported by the standard YAML format. This makes it easy to define arrays of complex data structures.

6

Simple lists of data are supported (a standard feature of YAML). The number of elements can be determined at runtime by the configuration manager.

All these qualities make the new configuration files very easy to read, maintain and extend.

3.7. Code repository and extensions

The GRASP code is managed by GIT. If you have a version that doesn't use GIT (for example, by downloading it from a web server), we strongly recommend you to look for a version downloaded via GIT (see Section 3.4, “Access to GRASP Open repository”). It will allow you to be connected to the server to make updates of the code. Additionally, the GRASP code can be extended in different parts. You can extend the input module or the output module. Extensions in input module are classified as drivers and transformers. A driver is a module that is called to read input data. The most basic driver is the SDATA driver which reads a SDATA file (Section 4.2.1, “The SDATA format”) but many other drivers can be implemented to read raw databases and inject, without using intermediate text files, data directly in the GRASP algorithm. A good example is to process satellite data, where the performance is a keystone to be able to process this kind of huge archives. Satellite data is not transformed to sdata format, instead of this, a specific driver is developed to connect raw satellite archive with GRASP. Transformers are another type of input extensions which allow to modify input data after reading it. An example of a transformer could be to load a climatology database to optimize input parameters. This action can be shared between different drivers and it is called after loading data by a driver. Output module can be extended with output functions. There are three types of output functions:

  • segment output function: It will be called after processing each segment of a tile
  • tile output function: It will be called after processing entire tile
  • current output function: It will be called after processing each segment, but it receives a partial tile as an argument. This function can print a tile with the current processed information

Each extension is distributed separately of the core code (except if an extension is considered as a core, for example, sdata driver is essential). To install a new extension you can place the source code in the specific place or you can use the grasp-manager, an extra tool which will help you to manage the code and its extensions. It is described below (Section 3.7.1, “GRASP Manager”), and here we will explain how to install manually an extension.

Extensions are the pieces of code that are detected and added during compilation. To install an extension, you have to place the code inside the core repository in a proper place and compile again. The corresponding places for each of the extensions are the following:

  • For input
    • drivers have to be placed in src/input/drivers
    • transformers have to be placed in src/input/transformers
  • For output
    • segment functions have to be placed in src/output/segment_functions
    • tile functions have to be placed in src/output/tile_functions
    • current functions have to be placed in src/output/current_functions
  • Kernels: They have to be placed in src/retrieval/internal_files
  • Constants sets: They have to be placed in src/retrieval/src/constants_sets

The following diagram shows how the GRASP repository integrates extension repositories inside itself.

Typing the command grasp without arguments, you will obtain information about how the software was compiled including available extensions. To know more details about the extensions and write one by your own, please go to technical documentation.

Trick: Since the GRASP code files are tracked by GIT it is not recommended to modify them, except if you want to develop something. If you want to place your tests and examples near the code, use a folder called "home" (it is created during compilation but you can create it by yourself, if you wish). This folder will not be tracked by GIT, allowing you to have your tests with the code without having conflicts with GIT.

3.7.1. GRASP Manager

GRASP Manager is a script placed in the GRASP root folder which simplify (even more) the update process. This script wraps Makefile system and adds a mechanism to work with GIT and the multi-repository environment used by GRASP (see Section 3.7, “Code repository and extensions”). Compilation of GRASP is always based on cmake. The make system wraps cmake, helping with the creation of a necessary folder structure, and grasp-manager script wraps make system, helping with the use of GIT, for users that don't feel comfortable using it. Also, it takes care of the extensions, downloading and installing them in the correct places. GRASP Manager script is configured via grasp-manager.yml file. Since it is a configuration file it is not tracked by GIT. Instead of this, a template called grasp-manager.yml.dist is offerred by the system. If there is no grasp-manager.yml file, it will be created as a copy of grasp-manager.yml.dist file, first time you run grasp-manager script.

To know the list of actions offered by this script, you can just type ./grasp-manager.sh. A list of available actions will be printed. In these sections we will explain the most interesting actions for the users, but there are more actions that can be interesting if you are a developer. Please remember to have a look at the technical documentation to know more about this script.

The basic actions for regular users are update-grasp and update-grasp-to-dev. They allow you to update the code and the extensions to the newest version. With update-grasp you will get the last stable version and with update-grasp-to-dev you will get the last unstable version (next code to be released), which is not recommended to use except if you know what you are doing ;-). These commands accept an argument that is called "environment". Environments are defined in grasp-manager.yml and allow you to customize the way to compile the code and the list of extensions that you want to have available. The documentation about how to write a configuration file for grasp-manager is written in the file grasp-manager.yml.dist. Please check that file, to know all available options. Note: Remember that in YAML format, lines that start by # symbol are comments.

When you run grasp-manager script, a backup of your code is created. This backup contains the information about the previous GIT commit and unsaved changes in the code. Please take into account that the repository has to be "clear" before performing any update actions, otherwise the changes will be undone and saved in a backup. You can find these backups in the home/grasp-manager folder. They are organized by time (when grasp-manager was executed) and you can apply them via rollback (applies last backup) or apply actions.

Finally, the next code will show an example of the use of grasp manager:

        
    
$ # Following command will move the repository to dev branch using the environment defined
$ # as release in grasp-manager.yml
$ # It will install/uninstall extension in order to have exactly the extensions release environment specify
$ ./grasp-manager.sh checkout dev release 
$ # Then, we can compile the code using the compilation settings defined in 'release' environment
$ # Since the release environment was the last used, it can be omitted.
$ # When the environment argument is omitted, we use the constants used
$ # in the last command, but we don't install/uninstall extensions (available extensions are already in use)
$ ./grasp-manager.sh make
$ # Finally we are going to install the code with the system in the last version.
$ # This is the most important action for regular users.
$ # A user of GRASP can regularly update the system via the following command:
$ ./grasp-manager.sh update-grasp 
                

3.8. Known problems

Listed here, are common problems found during installations process.
  • In MacOS system, some users receive fatal error: The remote end hung up unexpectedly; fatal: early EOF; fatal: index-pack failed

    It can be necessary to increase Git buffer size using the following command:git config --global http.postBuffer 1048576000



[2] For the distinction between the free and open source movements, see http://www.gnu.org/philosophy/free-software-for-freedom.en.html.

[3] That it is possible doesn't mean that it is recommended. In the current status of the project, GRASP has not been tested extensively with other solvers than SuperLU and the support for other solvers is still quite sloppy.