modelrunner.run.job module

Provides functions for submitting models as jobs

ensure_directory_exists(folder)

creates a folder if it not already exists

escape_string(obj)

escape a string for the command line

Return type:

str

get_config(config=None, *, load_user_config=True)

create the job configuration

Parameters:

• config (str or dict) – Configuration settings that will be used to update the default config

• load_user_config (bool) – Determines whether the file ~/.modelrunner is loaded as a YAML document to provide user-defined settings.

Returns:

the established configuration

Return type:

Config

get_job_name(base, args=None, length=7)

create a suitable job name

Parameters:

• base (str) – The stem of the job name

• args (dict) – Parameters to include in the job name

• length (int) – Length of the abbreviated parameter name

Returns:

A suitable job name

Return type:

str

submit_job(script, output=None, name='job', parameters=None, config=None, *, log_folder=None, method='auto', use_modelrunner=True, template=None, overwrite_strategy='error', **kwargs)

submit a script to the cluster queue

Parameters:

• script (str of Path) – Path to the script file, which contains the model

• output (str of Path) – Path to the output file, where all the results are saved

• name (str) – Name of the job

• parameters (str or dict) – Parameters for the script, either as a python dictionary or a string containing a JSON-encoded dictionary.

• config (str or dict) – Configuration for the job, which determines how the job is run. Can be either a python dictionary or a string containing a JSON-encoded dictionary.

• log_folder (str of Path) – Path to the logging folder. If omitted, the default of the template is used, which typically sends data to stdout for local scripts (which is thus captured and returned by this function) or writes log files to the current working directory for remote jobs.

• method (str) – Specifies the submission method. Currently background, foreground, ‘srun’, and qsub are supported. The special value auto reads the method from the config argument.

• use_modelrunner (bool) – If True, script is envoked with the modelrunner library, e.g. by calling python -m modelrunner {script}.

• template (str of Path) – Jinja template file for submission script. If omitted, a standard template is chosen based on the submission method.

• overwrite_strategy (str) – Determines what to do when files already exist. Possible options include error, warn_skip, silent_skip, overwrite, and silent_overwrite.

Returns:

The result (stdout, stderr) of the submission call. These two strings

can contain the output from the actual scripts that is run when log_folder is None.

Return type:

tuple

submit_jobs(script, output_folder, name_base='job', parameters=None, config=None, *, output_format='hdf', list_params=None, **kwargs)

submit many jobs of the same script with different parameters to the cluster

Parameters:

• script (str of Path) – Path to the script file, which contains the model

• output_folder (str of Path) – Path to the output folder, where all the results are saved

• name_base (str) – Base name of the job. An automatic name is generated on this basis.

• parameters (str or dict) – Parameters for the script, either as a python dictionary or a string containing a JSON-encoded dictionary. All combinations of parameter values that are iterable and not strings and not part of keep_list are submitted as separate jobs.

• config (str or dict) – Configuration for the job, which determines how the job is run. Can be either a python dictionary or a string containing a JSON-encoded dictionary.

• output_format (str) – File extension determining the output format

• list_params (list) – List of parameters that are meant to be lists. They will be submitted as individual parameters and not iterated over to produce multiple jobs.

• **kwargs – All additional parameters are forwarded to submit_job().

Returns:

The number of jobs that have been submitted

Return type:

int