Script of Scripts (SoS) consists of
SoS Notebook, a Jupyter-based polyglot notebook that allows the use of multiple Jupyter kernels in one notebook, and
SoS Workflow, a workflow system for the execution of workflows in both process- and outcome-oriented styles. It is designed for data scientists and bioinformatics who routinely work with scripts in different languages such as bash, Python, R, and SAS. This repository contains the
SoS Workflow workflow engine.
Please refer to relevant publications SoS Notebook: An Interactive Multi-Language Data Analysis Environment in Bioinformatics, Script of Scripts: a pragmatic workflow system for daily computational research in PLoS Computational Biology, and the SoS homepage for more information on SoS Notebook and SoS Workflow.
We welcome and value community contributions: please post issues to provide us feedback or get our support; please send pull requests if you have helped fixing bugs or making improvements to the source code. In addition, if you are interesting in extending SoS, for example adding a new target type or task engine, please have a look at our tutorial on extending SoS here.
If you are using conda, you can install sos using command
conda install sos -c conda-forge
You can also install a majority of the SoS suite of tools using commands such as
conda install sos sos-pbs sos-notebook jupyterlab-sos sos-bash sos-python sos-r -c conda-forge
This will install SoS Workflow, SoS Notebook and its JupyterLab extension, language modules for
R itself (
r-irkernel etc) and needed
feather) if needed.
If you are not using conda, you will have to install all the pieces one by one, but you will also have the freedom to use non-conda installation of
Julia etc. With a working Python 3.6 installation, you can install SoS Workflow with command
pip install sos
pip install sos-pbs
if you would like to use SoS with a remote batch system such as LSF or Slurm.
You can install SoS Notebook, and register the sos kernel with Jupyter using the following commands
pip install sos-notebook python -m sos_notebook.install
To exchange data between live kernels in SoS Notebook, you will need to install individual kernels (e.g.
R), make sure they work under Jupyter, and install SoS language modules using commands such as
pip install sos-r sos-matlab sos-python sos-bash
Different modules might be needed to assist data exchange among kernels. Please refer to the installation page of sos website for details.
SoS Notebook 0.22.3
SoS Notebook 0.22.2
pem_fileto authenticate to remote hosts.
-r hostto allow workflow to be executed with templates and on PBS
-b BIN_DIRdue to sos#1319
sos executeto use remote workers.
SoS Notebook 0.18.5
output_from(step)to obtain output from a workflow.
-Tto enfore dependency tracing.
tracedto make dependencies that will always be traced
dependsstatement always try to build dependency.
SoS Notebook 0.18.1
&to the end of magics
%runfileto execute workflow in background.
%rerunand add magic
%saveto execute the cell after saving it.
named_outputto support named inputs and outputs, among other new features summarized in this ticket.
paired_withetc for functions
concurrent=Trueas default for substep execution.
stop_ifand replace it with
sos_variableto import shared variable in a step
SoS Notebook 0.18.0
SoS 0.9.16.0, SoS Notebook 0.9.16.0
sos_step()target that matches multiple steps.
-pto generate a summary report after the completion of workflow.
sos run script -h).
SoS Notebook 0.9.14.4:
SoS Notebook 0.9.14.1
%revisionsmagic to display revision history of the current document.
SoS Notebook 0.9.13.4
%cdnow changes directory of all subkernels
SoS Notebook 0.9.12.12
zap()to SoS path classes
SoS Notebook 0.9.12.11
activeof actions and tasks to accept conditions.
~) for SoS path types
SoS Notebook 0.9.12.9
stderrto redict output from script-executing actions.
python -m sos_notebook.installto install
soskernel to Jupyter.
concurrent=Trueto allow parallel execution of input groups.
SoS Notebook 0.9.10.17
%captureto capture output of cells