parouby
/
Pamhyr2
mirror of https://gitlab.com/pamhyr/pamhyr2
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Pamhyr2/doc/dev/documentation.org

1422 lines
49 KiB
Org Mode
Raw Permalink Blame History

# documentation.org -- Pamhyr developers documentation
# Copyright (C) 2023-2024 INRAE
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
# -*- coding: utf-8 -*-
#+STARTUP: indent
#+INCLUDE: ../tools/macro.org
#+INCLUDE: ../tools/latex.org
#+TITLE: Developers documentation
#+SUBTITLE: Version: {{{version}}}
#+AUTHOR: {{{INRAE}}}
#+OPTIONS: toc:t
#+LANGUAGE: UKenglish
#+BEGIN_abstract
This document is for the use of developers. It describes the project
architecture, the tools available to assist development and
debugging. It also describes the procedures for creating packages and
the configurations required to set up the gitlab runners. Finally,
this document explains how documentation is written and modified, and
how to contribute to the project by modifying, improving or adding
documentation, translations or code.
#+END_abstract
* Introduction
{{{pamhyr2}}} is free and open source software (FOSS) graphical user
interface (GUI) for 1D hydro-sedimentary modelling of rivers developed
in Python (with version 3.8). It use PyQt at version 5 and matplotlib
in version 3.4.1 or later for the user insterface (see
{{{file(/requirements.txt)}}} for details). The architecture of
project code follow the Qt Model/View architecture [fn:qt-arch] (see
details in section [[Architecture]]). {{{pamhyr2}}} packages can be build
manually (see section [[Building packages]]), but there are automatically
build with the gitlab-ci (see the section [[Setup the CI
environment]]). Documentation files are written with org-mode[fn:org],
let see section [[Documentation files]]. Finally, to see the contribution
rules, see the section [[How to contribute?]].
[fn:qt-arch] Qt Model/View documentation:
https://doc.qt.io/qt-5/model-view-programming.html (last access
2023-09-15)
[fn:org] The org-mode website: https://orgmode.org/ (last access
2023-09-15)
* Architecture
{{{pamhyr2}}}'s architecture is based on Qt Model/View, see Figure
[[graph-architecture]]. It is made up of several different components: the
model (in blue), the graphical components (in red), the
actions/delegates (in green), the commands (in purple), the solvers
(in yellow) and the save file (in grey).
The model is a set of python classes and can be exported to a single
SQLite3 format backup file. The view can be made up of various
components, generally a Qt window with other view components, such as:
a table, a text box, a button, a plot, and so on. The user can view
the data using the view and interact with certain components. These
components are linked to an action (such as a Python function) or to a
delegate class. These actions or delegate can create a command (based
on Qt UndoCommand class), this command must implement two functions:
One to modify the model, one to reverte this modification and reset
the model to previous state. All model modification must be perform by
a command to be cancelled. The user can also run a solver and add some
simulation results to model data.
#+name: graph-architecture
#+header: :results drawer
#+header: :exports results
#+header: :post attr_wrap(width="12cm", data=*this*, name="graph-architecture", caption="Pamhyr2 Model/View architecture scheme (inspired by Qt Model/View architecture [[https://doc.qt.io/qt-5/model-view-programming.html]])", float="t")
#+begin_src dot :file "images/auto_graph-architecture.png" :cache no
digraph {
bgcolor="transparent";
node[colorscheme=set19,shape=box,style="filled",fillcolor=white];
edge[colorscheme=set19,color=0];
root[style=invis];
subgraph cluster0 {
label="File System"
style=dashed;
save[label="Pamhyr save",fillcolor="9",shape=note];
sbin[label="Solver binary",fillcolor="9",shape=note];
configfile[label="Pamhyr configuration file",fillcolor="9",shape=note];
}
user[label="User",shape=ellipse];
subgraph cluster1 {
label="Pamhyr2";
config[label="Configuration",fillcolor="5"];
model[label="Model",fillcolor="2"];
view[label="View",fillcolor="1"];
delegate[label="Delegate",fillcolor="3"];
action[label="Action",fillcolor="3"];
solver[label="Solver",fillcolor="6"];
undocommand[label="Command",fillcolor="4"];
}
root -> model[style=invis];
root -> config[style=invis];
model -> save[dir=both,label="Save/Load"];
config -> configfile[dir=both,label="Save/Load"];
undocommand -> solver[style=invis];
action -> solver[style=invis];
delegate -> solver[style=invis];
model -> view[label="Rendering"];
view -> delegate[label="Rendering"];
delegate -> undocommand[label="Create"];
action -> undocommand[label="Create/use"];
action -> solver[label="Run"];
solver -> model[dir=both,label="Export/Results",labelangle=0,labelfloat=true,constraint=true];
solver -> sbin[dir=both,label="Execute/Results"];
undocommand -> model[label="Modify"];
view -> user[label="Vizualize"];
user -> delegate[label="Modify"];
user -> action[label="Triggere"];
config -> solver[label="Create",style=dashed,labelfloat=true,constraint=true];
//model -> config[style=invis];
config -> model[style=invis];
delegate -> action[style=invis];
save -> configfile[style=invis];
root -> solver[style=invis];
user -> solver[style=invis];
}
#+end_src
All the model source code are in the directory {{{file(src/Model)}}}
(let see section [[Model]] for more details), the View components,
delegate and command are in {{{file(src/View)}}} (see section [[View]]). Solvers classes are
in {{{file(src/Solver)}}} (see section [[Solver]]).
The following sub section show examples of main {{{pamhyr}}} internal
class for view componants, but this documentation is not exhaustive,
be free to watch existing code for more details and examples. In,
addition some features are not factorise and must be implemented from
scratch (directly with Qt for example).
[fn:qt-mv] The Qt Model/View documentation web page:
https://doc.qt.io/qt-5/model-view-programming.html
** Model
The model is a set of Python classes. In {{{pamhyr2}}}, this classes must
respect some constraint. Each model class must inherits
=Model.Tools.SQLSubModel= abstract class, except the =Model.Study=
class who inherits =Model.Tools.SQLModel= (see [[SQL]]).
The model entry point is the Study class. It contains infomation about
the study: their name, description, time system, and so on. Their
contains a River object too. This river object inherits the network
graph and contains a list of =RiverNode= and a list of =RiverReach=
(an edge who contains a source node, and destination node).
=RiverReach= contrains geometry, so, the river network (node and edge)
associated with the geometry forms the basis of the model, and the
other components are linked to one of these basic components.
#+name: graph-model
#+header: :results drawer
#+header: :exports results
#+header: :post attr_wrap(width="16cm", data=*this*, name="graph-model", caption="Pamhyr2 model class dependencies (A -> B means A can contain references to B)", float="t")
#+begin_src dot :file "images/auto_graph-model.png" :cache no
digraph {
bgcolor="transparent";
node[colorscheme=set19,shape=box,style="filled",fillcolor="2"];
//subgraph cluster0 {
// style=dashed;
study[label="Study"];
river[label="River"];
subgraph cluster00 {
style=solid;
label="Network"
rnode[label="RiverNode"];
redge[label="RiverReach"];
}
subgraph cluster06 {
style=solid;
label="Greometry"
georeach[label="Reach"];
geocrosssection[label="Cross-section"];
geopoint[label="Point"];
}
//}
//subgraph cluster1 {
// style=dashed;
frictionlist[label="FrictionList"];
subgraph cluster01 {
style=solid;
label="Stricklers";
stricklers[label="Stricklers"];
stricklerslist[label="StricklersList"];
}
subgraph cluster02 {
style=solid;
label="BoundaryCondition";
boundaryconditionlist[label="BoundaryConditionList"];
boundarycondition[label="BoundaryCondition"];
}
subgraph cluster03 {
style=solid;
label="LateralContribution";
lateralcontributionlist[label="LateralContributionList"];
lateralcontribution[label="LateralContribution"];
}
subgraph cluster04 {
style=solid;
label="InitialConditions";
initialconditionsdict[label="InitialConditionsDict"];
initialconditions[label="InitialConditions"];
}
solverparameterslist[label="SolverParametersList"];
subgraph cluster05 {
style=solid;
label="Sediment";
sedimentlayerlist[label="SedimentLayerList"];
sedimentlayer[label="SedimentLayer"];
layer[label="Layer"];
}
//}
subgraph cluster2 {
style=dashed;
label="Results"
results[label="Results"]
rriver[label="River"];
rreach[label="Reach"];
rcrosssection[label="Cross-section"];
}
study -> river;
river -> rnode;
river -> redge;
redge -> rnode;
river -> boundaryconditionlist -> boundarycondition -> rnode;
river -> lateralcontributionlist -> lateralcontribution -> redge;
river -> initialconditionsdict -> initialconditions;
initialconditions -> redge;
river -> stricklerslist -> stricklers;
river -> solverparameterslist;
river -> sedimentlayerlist -> sedimentlayer -> layer;
redge -> frictionlist -> stricklers;
redge -> georeach -> geocrosssection -> geopoint;
geocrosssection -> sedimentlayer;
geopoint -> sedimentlayer;
results -> study;
results -> rriver;
rriver -> river;
rriver -> rreach;
rreach -> georeach;
rreach -> rcrosssection;
rcrosssection -> geocrosssection;
// river -> boundaryconditionlist -> boundarycondition -> results[style=invis];
// river -> lateralcontributionlist -> lateralcontribution -> results[style=invis];
// river -> initialconditionsdict -> initialconditions -> results[style=invis];
// initialconditions -> results[style=invis];
// river -> stricklerslist -> stricklers -> results[style=invis];
// river -> solverparameterslist -> results[style=invis];
// river -> sedimentlayerlist -> sedimentlayer -> layer -> results[style=invis];
geopoint -> boundaryconditionlist[style=invis];
geopoint -> lateralcontributionlist[style=invis];
geopoint -> initialconditionsdict[style=invis];
geopoint -> initialconditions[style=invis];
geopoint -> stricklerslist[style=invis];
geopoint -> solverparameterslist[style=invis];
geopoint -> sedimentlayerlist[style=invis];
}
#+end_src
*** SQL
The model must be export to a database file to create a study save
file. This file use SQLite3[fn:sqlite] format and the extention
=.pamhyr=. So, each model componante must be register into this study
file. To create, update, set and get information into SQLite database
we use SQL command. The database use version number and some
modification could be perform to update database. For each model
componante, correspond one or more SQL table to store information. To
normalize the interaction with database we made two classes, SQLModel
and SQLSubModel. The Study class use SQLModel because is the top of
the model hierachy. The rest of model class inherits to SQLSubModel.
A class who inherits SQLSubModel, must implement some methods:
- =_sql_create=: Class method to create the database scheme
- =_sql_update=: Class method to update the database scheme if necessary
- =_sql_load=: Class method to load data from DB
- =_sql_save=: Method to save current object into DB
Class method take in arguments: The class (=cls=), a function to
execute SQL command into the database (=execute=). In addition, the
update method take the previous version of database, load method take
an optional arguments =data= if additional infomation ar needed, and
who can contains whatever you want. The method save take in arguments
the current object (=self=), a function to execute SQL command into
the database (=execute=), and optional data (=data=).
The class who inherits SQLSubModel can also define an class attribute
=_sub_classes= to set a formal class dependencies into database. This
attribute is use at database creation to create all table, and at
update to update all the database table. Let see examples of
SQLSubModel usage for two classes Foo and Bar with Foo contains list
of Bar (Listing [[sql-bar]] and [[sql-foo]]).
#+NAME: sql-bar
#+CAPTION: Exemple of class Bar inherits SQLSubModel.
#+begin_src python :python python3 :results output :noweb yes
from Model.Tools.PamhyrDB import SQLSubModel
class Bar(SQLSubModel):
_id_cnt = 0
def __init__(self, id = -1, x = 0, y = 0):
self._x = x
self._y = y
if id == -1:
self.id = Bar._id_cnt + 1
else:
self.id = id
Bar._id_cnt = max(id, Bar._id_cnt+1)
@classmethod
def _sql_create(cls, execute):
execute("""
CREATE TABLE bar (
id INTEGER NOT NULL PRIMARY KEY,
x INTEGER NOT NULL,
y INTEGER NOT NULL,
foo_id INTEGER NOT NULL,
FOREIGN KEY(foo_id) REFERENCES foo(id),
)""")
return True
@classmethod
def _sql_update(cls, execute, version):
# If version is lesser than 0.0.2, add column to bar table
major, minor, release = version.strip().split(".")
if major == minor == "0":
if int(release) < 2:
execute("ALTER TABLE bar ADD COLUMN y INTEGER")
return True
@classmethod
def _sql_load(cls, execute, data = None):
new = []
table = execute(
f"SELECT id, x, y FROM bar WHERE foo_id = {data['id']}"
)
for row in table:
bar = cls(
id = row[0], x = row[1], y = row[2],
)
new.append(bar)
return new
def _sql_save(self, execute, data = None):
execute("INSERT INTO bar (id,x,y,foo_id) VALUES " +
f"({self.id}, {self._x}, {self._y}, {data['id']})")
#+end_src
#+NAME: sql-foo
#+CAPTION: Exemple of class Foo inherits SQLSubModel and contains a list of Bar object (Listing [[sql-bar]]).
#+begin_src python :python python3 :results output :noweb yes
class Foo(SQLSubModel):
_id_cnt = 0
_sub_classes = [Bar]
def __init__(self, id = -1, name = ""):
self._name = name
self._bar = []
# ...
@classmethod
def _sql_create(cls, execute):
execute("""
CREATE TABLE foo (
id INTEGER NOT NULL PRIMARY KEY,
name TEXT NOT NULL,
)
""")
return cls._create_submodel(execute)
@classmethod
def _sql_update(cls, excute, version):
return cls._update_submodel(execute, version)
@classmethod
def _sql_load(cls, execute, data = None):
new = []
table = execute(
"SELECT id, name FROM foo"
)
for row in table:
foo = cls(
id = row[0],
name = row[1],
)
data = {
"id": row[0], # Current Foo ID
}
foo._bar = Bar._sql_load(execute, data=data)
new.append(foo)
return new
def _sql_save(self, execute, data = None):
execute(f"DELETE FROM foo WHERE id = {self.id}")
execute(f"DELETE FROM bar WHERE foo_id = {self.id}")
# Save new data
execute(f"INSERT INTO bar (id,name) VALUES ({self.id}, {self._name})")
data = {"id": self.id}
for bar in self._bar:
bar._sql_save(execute, data=data)
#+end_src
Let see the results database scheme for {{{pamhyr2}}} at version v0.0.7 in
Figure [[sql_schema]].
#+NAME: sql_schema
#+ATTR_LATEX: :width 16cm
#+CAPTION: SQLite database scheme at {{{pamhyr2}}} version v0.0.7 (generate with [[https://gitlab.com/Screwtapello/sqlite-schema-diagram]])
[[./images/schema_v0.0.7.png]]
[fn:sqlite] The SQLite web site: https://www.sqlite.org/index.html
(last access 2023-09-20)
*** List class
A abstract class PamhyrModelList is available and provide some of
basic methods for object list in Model. This abstract class implement
some classic action in View like: insert new object, delete object,
sort, move object up, move object down, and so on. An variant exists
for multiple list with same type of object, each sublist is called
tab, because in View, this kind of list si prensented in different
table PamhyrModelListWithTab.
*** Dict class
A abstract class PamhyrModelDict is available and provide some of
basic methods for object dictionary in Model. This class is like
PamhyrModelList but use a dictionary instead of list.
** View
{{{pamhyr2}}} use Qt as graphical user interface library with the
application "Qt designer" for windows or widget creation (see [[UI file]])
and "Qt linguist" for interface translate (see [[Translate]]). In
addition, we use matplotlib as ploting library (see [[Plot]]).
Typically, each model componant have an associated window in
application to add, delete or edit this componant. At top level of
View directory we found the =MainWindow.py= file and some
sub-directories. A view sub-directory contains: A =Window.py= file, a
=Table.py= file with table model definition if nessessary, one or more
=Plot*.py= file with plot class definition, a =translate.py= file with
componant translate, and possible other files or sub-directories.
*** UI file
We define as possible all {{{pamhyr2}}} windows and custom widgets
with "Qt designer". This application generate UI file who describes
interface organisation with table, layout, button, etc. This method is
faster than hand made windows and widget creation, and saves us some
purely descriptive code. The UI files are saved into =src/View/ui= for
window, and =/src/View/ui/Widgets= for custom widget.
*** Translate
#+NAME: pamhyr-trad
#+CAPTION: Example of =PamhyrTranslate= class implementation with a global traduction for /FooBar/ and a additional dictionary =table_headers=
#+begin_src python :python python3 :results output :noweb yes
from PyQt5.QtCore import QCoreApplication
from View.Tools.PamhyrTranslate import PamhyrTranslate
_translate = QCoreApplication.translate
class MyTranslate(PamhyrTranslate):
def __init__(self):
super(MyTranslate, self).__init__()
# Add traduction to global dictionary
self._dict["My"] = _translate("My", "FooBar")
# Add an additional translate dictionary
self._sub_dict["table_headers"] = {
"foo": _translate("My", "Foo"),
"bar": _translate("My", "Bar"),
"baz": _translate("My", "Baz"),
}
#+end_src
*** Window
The abstract class =PamhyrWindow= and =PamhyrDialog= are used for most
of {{{pamhyr2}}} window. These class allow to create an window for
{{{pamhyr2}}} GUI and implemente some useful methods. The super class
method difine some generic value from optional parameters, for
examples:
- =self._study=: The study giving in constructor parameters =study=
(typically a =Model.Study= class object)
- =self._config=: The configuration giving in constructor parameters =config= (typically a =Config= class object)
- =self._trad=: The traductor dictionary giving in constructor
parameters =trad= (typically a =Model.Tools.PamhyrTranslate= class
object)
#+NAME: window
#+CAPTION: Example of {{{pamhyr2}}} window
#+begin_src python :python python3 :results output :noweb yes
from View.Tools.PamhyrWindow import PamhyrWindow
from View.My.Translate import MyTranslate
from View.My.Table import MyTableModel
class MyWindow(PamhyrWindow):
_pamhyr_ui = "MyUI"
_pamhyr_name = "My window"
def __init__(self, study=None, config=None,
my_data=None,
parent=None):
self._my_data = my_data
super(MyWindow, self).__init__(
# Window title
title = self._pamhyr_name + " - " + study.name,
# Window standard data
study = study, config = config,
trad = MyTranslate(),
parent = parent,
# Activate undo/redo and copy/paste shortcut
options = ["undo", "copy"]
)
# Add custom data to hash window computation
self._hash_data.append(self._my_data)
# Setup custom window components
self.setup_table()
self.setup_connections()
def setup_table(self):
# Init table(s)...
def setup_connections(self):
# Init action connection(s)...
# ...
#+end_src
Typically we called method =setup_*=, the method to initialize some
window componants or connections.
*** Table
An abstract class =PamhyrTableModel= is available to define a simple
QAbstractTableModel shortly. In simple cases, there are only =data=
and =setData= methode to implement, but the constructor needs more
information than a classic QAbstractTableModel class.
#+NAME: table-src
#+CAPTION: Definition of a table model from =PamhyrTableModel= in a file =View/My/Table.py=.
#+begin_src python :python python3 :results output :noweb yes
from View.Tools.PamhyrTable import PamhyrTableModel
class MyTableModel(PamhyrTableModel):
def data(self, index, role):
# Retrun data at INDEX...
@pyqtSlot()
def setData(self, index, value, role=Qt.EditRole):
# Set VALUE at INDEX...
#+end_src
#+CAPTION: Using the table model defined in Listing [[table-src]] in window funtion =setup_table= defined Listing [[window]].
#+begin_src python :python python3 :results output :noweb yes
# Table model creation (Window.py: setup_table)
table_headers = self._trad.get_dict("table_headers")
self._model = MyTableModel(
table_view = table, # The table view object
table_headers = table_headers, # The table column headers dict
# (with traduction)
editable_headers = ["foo", "bar"], # List of editable column name
delegates = {
"bar": self.my_delegate, # Custom delegate for column 'bar'
},
data = self._my_lst, # The data
undo = self._undo_stack, # The window undo command stack
)
#+end_src
*** UndoCommand
All model modification must be done by an QUndoCommand, this command
allow to undo and redo an action. This a Qt class wi can inherit to
define custom undo command (see example Listing [[undo-cmd]])
#+NAME: undo-cmd
#+CAPTION: Example of custom UndoCommand, this command allow to add a node to graph in river network window (method redo), and delete it to graph with undo method
#+begin_src python :python python3 :results output :noweb yes
class AddNodeCommand(QUndoCommand):
def __init__(self, graph, node):
QUndoCommand.__init__(self)
self._graph = graph
self._node = node
def undo(self):
self._graph.remove_node(self._node)
def redo(self):
self._graph.insert_node(self._node)
#+end_src
All undo command must be push into a =QUndoStack= (see Listing
[[undo-cmd-push]]) to perform the action and allow user undo and redo this
action. In =PamhyrWindow= (and =PamhyrDialog=) the undo stack is
automatically create if the option ="undo"= is activate at window
creation, this stack is accessible at =self._undo_stack=.
#+NAME: undo-cmd-push
#+CAPTION: Example of UndoCommand push into an undo stack.
#+begin_src python :python python3 :results output :noweb yes
self._undo_stack.push(
AddNodeCommand(
self._graph,
node
)
)
#+end_src
*** Plot
To define a new plot you can create a class who inherit to
=PamhyrPlot=. The creator need at leaste five argument:
- A =canvas= of type =MplCanvas=
- A (optional) =trad= of type =PamhyrTranslate=
- A =data= used in =draw= and =update= to create and update the plot
- A optional =toolbar= of type =PamhyrToolbar=
- A =parent= window
This class must implement two method =draw= and =update=, the first
method to draw the plot from scratch, the second to update the plot if
data has changed.
#+begin_src python :python python3 :results output :noweb yes
from View.Tools.PamhyrPlot import PamhyrPlot
class MyPlot(PamhyrPlot):
def __init__(self, canvas=None, trad=None, toolbar=None
data=None, parent=None):
super(MyPlot, self).__init__(
canvas=canvas,
trad=trad,
data=data,
toolbar=toolbar,
parent=parent
)
self.label_x = self._trad["x"]
self.label_y = self._trad["y"]
# Optional configuration
self._isometric_axis = False
self._auto_relim_update = True
self._autoscale_update = True
def draw(self):
# Draw function code...
def update(self):
# Update function code...
def clear(self):
# Clear plot values...
# ...
#+end_src
** Solver
The {{{pamhyr2}}} architecture allow to define multiple solver. A
solver is define by a:
- type
- name
- description,
- path
- command line pattern
- (optional) input formater path
- (optional) input formater command line
- (optional) output formater path
- (optional) output formater command line
Let see Figure [[graph-multi-solver]], the application can implement
different solver type, this solver type implement the code for export
study to solver input format, and read the solver output to study
results. There exists a generic solver with a generic input and output
format, the type could be use to use a solver not implemented in
{{{pamhyr2}}}, but this solver must can read/write input and output
generic format or use external script. There is possible to define
different solver with the same type, for example two differents
version of the same solver. Finaly, with input and output formater is
possible to execute a code on distant computer, for example, over ssh.
#+name: graph-multi-solver
#+header: :results drawer
#+header: :exports results
#+header: :post attr_wrap(width="12cm", data=*this*, name="graph-multi-solver", caption="Scheme of multiple solver configured, one Rubarbe solver and two Mage solver with one on local machine and one on a distant machine accessed over ssh", float="t")
#+begin_src dot :file "images/auto_graph-multi-solver.png" :cache no
digraph {
bgcolor="transparent";
node[colorscheme=set19,shape=box,style="filled",fillcolor=9];
edge[colorscheme=set19,color=0];
subgraph cluster00 {
label="User personal computer";
style=solid;
subgraph cluster0 {
label="Pamhyr2";
style=solid;
// subgraph cluster01 {
// label="Core";
//model[label="Model", fillcolor=2];
//view[label="View", fillcolor=1];
config[label="Configuration", fillcolor=5];
// view -> model -> view;
// }
subgraph cluster02 {
label="Solver";
style=dashed;
subgraph cluster021 {
label="Solver Classes";
//classSolverM7[label="Mage7", fillcolor=6];
classSolverM8[label="Mage8", fillcolor=6];
classSolverR[label="RubarBE", fillcolor=6];
}
//classSolverX[label="Solver X Binding", fillcolor=6];
subgraph cluster022 {
label="Solver Object";
solverM[label="Mage", fillcolor=6];
solverM2[label="Mage over ssh", fillcolor=6];
solverR[label="RubarBE", fillcolor=6];
//solverX[label="Solver X", fillcolor=6];
}
classSolverM8 -> solverM [style=dashed];
classSolverM8 -> solverM2[style=dashed];
classSolverR -> solverR[style=dashed];
//classSolverX -> solverX[style=dashed];
}
//config -> solverM[style="dotted"];
//config -> solverR[style="dotted"];
//config -> solverX[style="dotted"];
//model -> solverM;
//model -> solverM2;
//model -> solverR;
//model -> solverX;
}
subgraph cluster1 {
label="File System";
style=dashed;
mage[label="Mage Binary",shape=note];
//X[label="Solver X Binary"];
rubarbe[label="RubarBE Binary",shape=note];
ssh[label="ssh",shape=note];
}
}
//config -> X[style=invis];
//model -> config[style=invis];
config -> solverM[label="",constraint=true];
//config -> solverX[label="",constraint=true];
config -> solverR[label="",constraint=true];
config -> solverM2[label="",constraint=true];
subgraph cluster2 {
label="Distant server";
style=solid;
sshd[label="sshd"];
subgraph cluster21 {
label="File System";
style=dashed;
mage2[label="Mage Binary",shape=note];
}
}
solverM -> mage[label="", color=1];
mage -> solverM[label="", color=2];
//solverX -> X[label="", color=1];
//X -> solverX[label="", color=2];
solverR -> rubarbe[label="", color=1];
rubarbe -> solverR[label="", color=2];
solverM2 -> ssh -> sshd -> mage2[label="", color=1];
mage2 -> sshd -> ssh -> solverM2[label="", color=2];
}
#+end_src
Let see Figure [[graph-pipeline]] the temporal order of action to run a
solver and get results:
- (1) Write solver input file(s) using the study data
- (2) Run the solver
- (2.1) The solver read the input file(s)
- (2.2) The solver compute results and write it to solver output
file(s)
- (3) {{{pamhyr2}}} create a =Results= object
- (3.1) The {{{pamhyr2}}} solver class read solver output file(s) and
complete Results with readed data
#+name: graph-pipeline
#+header: :results drawer
#+header: :exports results
#+header: :post attr_wrap(width="10cm", data=*this*, name="graph-pipeline", caption="Pamhyr2 solver execution pipeline architecture scheme", float="t")
#+begin_src dot :file "images/auto_graph-pipeline.png" :cache no
digraph {
bgcolor="transparent";
node[colorscheme=set19,shape=box,style="filled",fillcolor=9];
edge[colorscheme=set19,color=0];
subgraph cluster0 {
label="Pamhyr2"
config[label="Configuration",fillcolor=5];
model[label="Model",fillcolor=2];
obj[label="Solver",fillcolor=6];
results[label="Results",fillcolor=2];
view[label="View",fillcolor=1];
results -> model[style="dashed"];
results -> obj[style="dashed"];
}
config -> obj[label=""];
obj -> model[style="dashed"];
subgraph cluster1{
label="File System";
style=dashed;
in[label="Solver input file(s)",shape=note,fillcolor=white];
out[label="Solver output file(s)",shape=note,fillcolor=white];
bin[label="Solver binary",shape=note];
}
obj -> in[label="Write (1)",color=1];
obj -> bin[label="Execute (2)",color=1];
bin -> in[label="Read (2.1)",color=1];
bin -> out[label="Write (2.2)",color=2];
obj -> results[label="Create (3)",color=2];
obj -> out[label="Read (3.1)", color=2];
view -> model[style="dashed"];
view -> results[style="dashed"];
}
#+end_src
In case of generic solver (or a solver with input and output formater)
the temporal order of action is prensented in Figure
[[graph-pipeline-generic]].
#+name: graph-pipeline-generic
#+header: :results drawer
#+header: :exports results
#+header: :post attr_wrap(width="12cm", data=*this*, name="graph-pipeline-generic", caption="Pamhyr2 generic solver execution pipeline architecture scheme", float="t")
#+begin_src dot :file "images/auto_graph-pipeline-generic.png" :cache no
digraph {
bgcolor="transparent";
node[colorscheme=set19,shape=box,style="filled",fillcolor=9];
edge[colorscheme=set19,color=0];
subgraph cluster0 {
label="Pamhyr2"
config[label="Configuration",fillcolor=5];
model[label="Model",fillcolor=2];
obj[label="Generic solver",fillcolor=6];
results[label="Results",fillcolor=2];
view[label="View",fillcolor=1];
results -> model[style="dashed"];
results -> obj[style="dashed"];
}
config -> obj[label=""];
obj -> model[style="dashed"];
subgraph cluster1{
label="File System";
style=dashed;
gin[label="Generic input file", shape=note,fillcolor=white];
ibin[label="Input formater", shape=note];
in[label="Solver input file(s)",shape=note,fillcolor=white];
out[label="Solver output file(s)",shape=note,fillcolor=white];
gout[label="Generic results file",shape=note,fillcolor=white];
obin[label="Output formater", shape=note];
bin[label="Solver binary",shape=note];
}
gin -> ibin[style=invis];
ibin -> bin -> obin[style=invis];
in -> bin[style=invis];
obin -> gout[style=invis];
// Input format
obj -> gin[label="Write (1)",color=1];
obj -> ibin[label="Execute (2)",color=1,style=dashed];
ibin -> gin[label="Read (2.1)",color=1];
ibin -> in[label="Write (2.2)",color=1];
// Solve
obj -> bin[label="Execute (3)",color=1,style=dashed];
bin -> in[label="Read (3.1)",color=1];
bin -> out[label="Write (3.2)",color=2];
// Output format
obj -> obin[label="Execute (4)",color=2,style=dashed];
obin -> out[label="Read (4.1)",color=2];
obin -> gout[label="Write (4.2)",color=2];
// Read results
obj -> results[label="Create (5)",color=2];
obj -> gout[label="Read (5.1)", color=2];
view -> model[style="dashed"];
view -> results[style="dashed"];
}
#+end_src
To implement a Solver in {{{pamhyr2}}}, there exists a abstract class
=Solver.AbstractSolver=. A class who herits this class, must implement
different methods:
- =export=: Export the study to solver input file(s)
- =input_param=: Return the solver input parameter(s) as string
- =log_file=: Return the solver log file name as string
- =results=: Read the solver output file(s) and return a =Model.Results= object.
** Unit tests
A very small part of {{{pamhyr2}}} has unit test. This part is limited to the Model.
#+begin_src shell
python3 -m venv test
. test test/bin/activate
pip3 install -U -r ./full-requirements.txt
cd src/
python3 -Walways -m unittest discovert -v -t .
#+end_src
** The debug mode
To activate an deactivate the {{{pamhyr2}}} debug mode you can open
the configuration window and type "Ctrl+G" or run {{{pamhyr2}}} with
command line:
#+begin_src shell
./Pamhyr2 debug
#+end_src
This mode add some log and add two action in main window menu: "About
> Debug" open a window with Python Repl in current Python
environement, and "About > Debug SQLite" who open the application
SQLiteBrowser (if installed) on current Study to explore the study
data base file.
#+NAME: debug-repl
#+ATTR_LATEX: :width 14cm
#+CAPTION: {{{pamhyr2}}} debug Python REPL
[[./images/python-debug-repl.png]]
* Build the project
The project uses gitlab-ci runners to build packages, but it is possible
to build packages manually.
** Building packages
If you need an hand made package, you can script available in
{{{file(packages)}}} directory.
*** GNU/Linux {{{linux}}}
On GNU/Linux building GNU/Linux packages is easy, you just need python
in version 3.8 must be installed with venv and pyinstaller packages
(see Listing [[linux-env-deb]] for Debian and derived system). Finally,
run the {{{file(linux.sh)}}} script (see Listing [[linux-pkg]]).
#+NAME: linux-env-deb
#+CAPTION: Install environment on GNU/Linux
#+begin_src shell
sudo apt install python3.8
python3 -m pip install venv
python3 -m pip install pyinstaller
#+end_src
#+NAME: linux-pkg
#+CAPTION: Build GNU/Linux package
#+begin_src shell
cd packages
./linux.sh
#+end_src
*** Windows {{{windows}}}
To make the Windows packages you have two choice: If you use Windows
you can use the script {{{file(packages/windows.bat)}}}, other else
you can use the script {{{file(packages/wine.sh)}}}. Each script need
a specific software environment.
On windows, you needs python on version 3.8, pyinstaller and
NSIS[fn:nsis] installed. On GNU/Linux you need wget, wine and
winetricks installed.
[fn:nsis] The NSIS web site: https://sourceforge.net/projects/nsis/
** Setup the CI environment
{{{pamhyr2}}} need a Linux ci-runner and a Windows ci-runner for building
package. The windows ci-runner could run on a Wine environement.
*** Linux {{{linux}}}
The Linux ci-runner need some software and dependencies in addtion of
gitlab-ci.
#+begin_src shell
sudo apt install \
emacs emacs-goodies-el \
texlive-full \
python3.8 python3.8-venv
sudo python3 -m pip install pyinstaller
#+end_src
*** Windows (Wine) {{{windows}}}
The ci-runner environment for Wine need at least wine version 8, let
[[https://www.numetopia.fr/comment-installer-wine-sur-ubuntu-ou-linux-mint/][see who to add wine official depot to your linux distribution]].
#+begin_src shell
sudo apt install wine-stable winetricks
#+end_src
**** Setup environment
Export Wine environment variable to set wine as 64 bits architecture
and set the correct path for wine environment.
#+begin_src shell
export WINARCH=win64
export WINEPREFIX=$PWD/my-wine-runner-prefix
#+end_src
Setup Wine environment to Windows 10 and install the minimal fonts
with =winetricks=.
#+begin_src shell
winetricks corefonts win10
#+end_src
**** Install dependencies
First install 7zip with help of =winetricks=.
#+begin_src shell
winetricks 7zip
#+end_src
In addition, install in the environment the Windows version of:
- [[https://www.python.org/ftp/python/3.8.10/python-3.8.10-amd64.exe][Python 3.8.10]] (ensure the python path is set and Pip is enable)
- [[https://git-scm.com/downloads][Git]]
- [[https://github.com/PowerShell/PowerShell/releases/download/v7.0.1/PowerShell-7.0.1-win-x64.msi][PowerShell]]
- [[https://freefr.dl.sourceforge.net/project/nsis/NSIS%203/3.08/nsis-3.08-setup.exe][Nsis]]
To run a Windows executable into wine environement, use =wine64= command:
#+begin_src shell
wine64 <the-exe-file>
#+end_src
Now, we can install =pyinstaller= on this windows environment:
#+begin_src shell
wine python -m pip install pyinstaller
#+end_src
Now, we can download [[https://docs.gitlab.com/runner/install/windows.html][Gitlab-ci]] runner for Windows an put it in the
current path.
**** Setup runner
You can configure the runner with command:
#+begin_src shell
wine64 gitlab-runner-windows-amd64.exe register
#+end_src
**** Run the runner
Create a new executable shell script =runner.sh= with following lines:
#+begin_src shell
#! /bin/sh
export WINARCH=win64
export WINEPREFIX=$PWD/my-wine-runner-prefix
wine64 gitlab-runner-windows-amd64.exe run
#+end_src
Now you can run the runner with command =./runner.sh=.
{{{bulb}}} You can run this command into a =screen= terminal, detach
the terminal and disconnect from runner machine to keep runner alive.
* Documentation files
This document and the user documentation are org files. This text file
format is formatted so that it can be exported in different formats:
PDF (with latex), ODT, HTML, etc. It was originally designed for the
GNUEmacs[fn:emacs] text editor, but can be edited with any text editor. Here we
take a look at the different features used in these documents.
[fn:org] The org-mode website: https://orgmode.org/ (last access
2023-09-15)
[fn:emacs] The GNUEmacs project website: https://gnu.org/s/emacs/
(last access 2023-09-15)
** Org-mode
*** Document structure
Org uses the =*= character to define a new document section. To add a
sub-section, you can add an additional =*= to the current section[fn::
See document structure documentation:
https://orgmode.org/org.html#Headlines (last access 2023-09-15)].
#+BEGIN_EXAMPLE
* Top level headline
** Second level
*** Third level
some text
*** Third level
more text
* Another top level headline
#+END_EXAMPLE
*** Format
Org-mode is a markup file, using markup in the text to modify the
appearance of a portion of text[fn:: See markup documentation:
https://orgmode.org/org.html#Emphasis-and-Monospace (last access
2023-09-15)].
| Markup | Results |
|--------------------+------------------|
| =*Bolt*= | *Bolt* |
| =/Italic/= | /Italic/ |
| =_underline_= | _underline_ |
| ==verbatim== | =verbatim= |
| =~code~= | ~code~ |
| =+strike-through+= | +strike-through+ |
*** Source code blocks
You can add some code blocks[fn:: See org-mode documentation for
source code: https://orgmode.org/org.html#Working-with-Source-Code
(last access 2023-09-15)] in the document.
Here is an example for python source code:
#+BEGIN_EXAMPLE
#+CAPTION: Get os type name in Python code
#+begin_src python
import os
print(f"Document build on system: {os.name}")
#+end_src
#+END_EXAMPLE
If you use GNUEmacs, it is also possible to run the code inside a
block and export (or not) the reuslts in the document.
#+OPTIONS: float:nil
#+CAPTION: Get os type name in Python code
#+begin_src python :python python3 :results output :exports both :noweb yes
import os
print(f"Document build on system: {os.name}")
#+end_src
#+RESULTS:
: Document build on system: posix
*** LaTeX
If we export the file to PDF, org-mode use {{{latex}}}. So we can add
some piece of {{{latex}}} into the document[fn:: See {{{latex}}} part
in documentation: https://orgmode.org/org.html#Embedded-LaTeX (last
access 2023-09-15)]. For exemple, we can add math formula like
=$E=mc^2$= ($E=mc^2$) or =\[E=mc^2\]=:
\[E=mc^2\]
But we can also add every type of {{{latex}}}:
#+BEGIN_EXAMPLE
# Add latex in line
#+LATEX: <my line of latex>
# Add multiple line of LaTeX
#+BEGIN_EXPORT latex
<my latex here>
#+END_EXPORT
#+END_EXAMPLE
It is also possible to add specific {{{latex}}} file header with
=#+LATEX_HEADER=. In this document we use the file
{{{file(doc/tools/latex.org)}}} for all {{{latex}}} headers.
*** Macro
In this document, we use a few macros[fn:: See marcos documentation
https://orgmode.org/org.html#Macro-Replacement (last access
2023-09-15)] to simplify writing. They allow you to define sequences
of text to be replaced, so that the macro name is replaced by its
value. They are defined in the {{{file(doc/tools/macro.org)}}}
file. Once defined, they can be used in the document as follows:
={{{<macro-name>}}}=. You can also have macros with arguments, in this
case: ={{{<macro-name>(arg1,...)}}}=. Les macros peuvent aussi
utiliser du code emacs-lisp.
#+BEGIN_EXAMPLE
# Exemple of macro définition
#+MACRO: toto tata
#+MACRO: add \(($1 + $2)\)
#+MACRO: emacs-version (eval (nth 2 (split-string (emacs-version))))
#+END_EXAMPLE
#+MACRO: toto tata
#+MACRO: add \(($1 + $2)\)
#+MACRO: emacs-version (eval (nth 2 (split-string (emacs-version))))
Macro apply:
- Marco ={{{toto}}}=: {{{toto}}}
- Marco ={{{add(x,y)}}}=: {{{add(x,y)}}}
- Marco ={{{emacs-version}}}=: {{{emacs-version}}}
*** Footnotes
Footnote in org-mode is define with marker =[fn:...]=[fn:: Create
footnotes in org-mode documentation
https://orgmode.org/org.html#Creating-Footnotes (last access
2023-09-15)]:
#+BEGIN_EXAMPLE
The Org website[fn:1] now looks a lot better than it used to.
...
[fn:1] The link is: https://orgmode.org
#+END_EXAMPLE
or:
#+BEGIN_EXAMPLE
The Org website[fn:: The link is: https://orgmode.org] now looks
a lot better than it used to.
...
#+END_EXAMPLE
*** References
The references use the {{{latex}}} bibtex tools. The bib file is in
{{{file(/doc/tools/ref.bib)}}} and use for developers and user
documentation. In document, use ={{{cite(<name>)}}}= to cite a paper.
** Export
To export the files, a {{{file(build.sh)}}} script is available in the org
files directories. On GNU/Linux system you can build the documentation
PDF file with the command =./build.sh=. Texlive package must be
installed, you can install only needed packages or all texlive
packages, for example on Debian (and some derived system) use command
Listing [[texlive-install]].
#+NAME: texlive-install
#+CAPTION: Installation command for texlive full on Debian system
#+begin_src shell
sudo apt install texlive-full
#+end_src
Some org-mode configuration used in documentations files are define in
=/doc/tools/=:
- {{{file(PamhyrDoc.cls)}}}: The {{{latex}}} document class
- {{{file(macro.org)}}}: Available macro
- {{{file(latex.org)}}}: {{{latex}}} configutation for documentations
files
- {{{file(setup.el)}}}: GNUEmacs configuration to build documentations
- {{{file(ref.bib)}}}: Bibtex files for documentations files
* How to contribute?
{{{pamhyr2}}} is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License[fn:license],
either version 3 of the License, or any later version.
[fn:license] The GPLv3 web page:
https://www.gnu.org/licenses/gpl-3.0.en.html
** Guidelines
To contribute to {{{pamhyr2}}}, we expect a minimum of respect between
contributors. We therefore ask you to respect the following rules
regarding communication and contribution content:
+ No gender, racial, religious or social discrimination
+ No insults, personal attacks or potentially offensive remarks
+ {{{pamhyr2}}} is free software, and intended to remain so, so take
care with the licensing of libraries and external content you want
to add to the project
+ Humour or hidden easter eggs are welcome if they respect the
previous rules
** Make a contribution
There are several ways to contribute: you can report a bug by creating
an issue on the project's gitlab page[fn:p2-gitlab], or you can create
a merge request on the same page with the changes you have made to the
code, translation or documentation.
The {{{pamhyr2}}} copyright is owned by INRAE[fn:inrae], but we keep a
record of each contributors. If you made a modification to
{{{pamhyr2}}} software, please add your name at the end of
{{{file(AUTHORS)}}} file and respect the Listing [[auth-format]]
format. You can update this file information for following
contribution.
#+NAME: auth-format
#+CAPTION: =AUTHORS= file format
#+begin_src text
<first name> <last name> [(optional) email], <organisation>, <years>
#+end_src
#+CAPTION: Current =AUTHORS= file
#+INCLUDE: "../../AUTHORS" src text
[fn:p2-gitlab] The {{{pamhyr2}}} Gitlab project page:
https://gitlab.com/pamhyr/pamhyr2
[fn:inrae] The INRAE web site: https://www.inrae.fr/
** Translate
You can improve or add translation for the project. To contribute to
{{{pamhyr2}}} translate, you need to use Qt
Linguist[fn:qt-linguist]. Open Qt-linguist and edit the translation
({{{file(.ts)}}}) file, finally, commit the new version of file and
make a merge request.
If you want add a new language, edit the script
{{{file(src/lang/create_ts.sh)}}} like Listing [[ts-it]]. Run the script
and open the new file with Qt-linguist, setup target language (Figure
[[qt-linguist-setup]]) and complete translation. Finally, commit the new
file and make a merge request.
#+NAME: ts-it
#+CAPTION: Example of modified {{{file(src/lang/create_ts.sh)}}} to add italian (it) translate for {{{pamhyr2}}}
#+begin_src shell
...
LANG="fr it"
...
#+end_src
#+NAME: qt-linguist-setup
#+ATTR_LATEX: :width 8cm
#+CAPTION: Qt linguist lang setup example with italian.
[[./images/Qt-linguist-setup-lang.png]]
[fn:qt-linguist] The Qt linguist documentation web page:
https://doc.qt.io/qt-5/qtlinguist-index.html (last access 2023-09-18)
** Code contribution
If you are developper you can improve and/or add features to
{{{pamhyr2}}}. Please, follow the architecture described in section
[[Architecture]] as closely as possible. Keep the code simple, clear and
efficient as possible. The master branch is reserved for the project
maintainer; you can create a new branch or fork the project before the
request.
{{{biblio}}}
Reference in New Issue View Git Blame Copy Permalink