What Are RAIL Stages?

A RAIL stage is a single algorithm that can be run in a reproducible way. Each stage has a specific goal, but can be configured through the use of parameters. They are the basic building blocks of a larger pipeline and can be parallelized across processors or computing nodes.

Types of Stages

There are three main types of stages in RAIL:

  1. Creation: create sample photometric data

  2. Estimation: estimate photometric redshift from any input data

  3. Evaluation: evaluate the performance of estimations against known true values

  4. Tools: “pseudo-stage” providing utilities and tools for running stages

These stages generate data products that can be used independently, or as input for other stages.

../../_images/stages_detailed.png

Examples

Here we link the RAIL stages as listed above, with how they can be used to answer specific scientific questions.

Comparing Redshift Algorithms

As one example, you may wish to examine which method of calculating photometric redshifts is most appropriate for your dataset and final use case.

To do this with RAIL, you would:

  1. Gather the data (either from an external source, or by generating it with RAIL)

  2. Use several different estimation algorithms (stages available under the rail.estimation.algos namespace), to estimate photometric redshift values for the galaxies.

  3. Evaluate the photometric redshift values, either by your own methods, or by using the evaluators and available metrics from the rail.evaluation namepace.

For a worked example of a case like this, see the Goldenspike notebook.

Comparing Data Degradation Algorithms

As another example, you may wish to find what kind of data is best estimated by a given redshift estimation algorithm.

To do this with RAIL, you would:

  1. Select an estimation algorithm and the evaluator stage that assesses its performance.

  2. Use several different creator and degrader algorithms to determine which data creation and noise / bias algorithms generate datasets best estimated by the selected estimation algorithm.

For a worked example of a case like this, see Exploring the Effects of Degraders which explores how differently degraded training data sets affect how well the estimation algorithm works.

Methods of Running RAIL

Interactive Mode

RAIL stages can be run interactively, such as in a Jupyter notebook. This is done via specific function calls.

To learn about the available functions, visit the Interactive API.

To see examples of using stages interactively, visit the Interactive Mode Notebooks.

Pipeline Mode

rail.core.RailStage is the main class. It serves as the base class for all the classes that implement parts of an analysis, and sets of RailStage can be combined into a rail.core.RailPipeline to perform a complete analysis.

In short, any RailStage does a single operation in a reproducible way. It can be thought of as a way to wrap a single function so as to make it both configurable and reproducible.

This presents a few constraints and limitations: the RailStage needs to save the configuration of any parameters it uses, and it needs to get input from as well as write output to specified locations. While the RailStage class provides mechanism to do all of these things, sub-classes must implement the actual computational work of the stage.

Below we give the API documentation for a few key elements of a RailStage.

RailStage basics

class rail.core.RailStage

Base class for rail stages

This inherits from ceci.PipelineStage and implements rail-specific data handling In particular, this provides some very useful features:

1. Access to the DataStore, which keeps track of the various data used in a pipeline, and provides access to each by a unique key.

2. Functionality to help manage multiple instances of a particular class of stage. The original ceci design didn’t have a mechanism to handle this. If you tried you would run into name clashes between the different instances. In ceci 1.7 we added functionality to ceci to allow you to have multiple instances of a single class, in particular we distinguish between the class name (cls.name) and and the name of the particular instance (self.instance_name) and added aliasing for inputs and outputs, so that different instances of PipelineStage would be able to give different names to their inputs and outputs. However, using that functionality in a consistent way requires a bit of care. So here we are providing methods to do that, and to do it in a way that uses the DataStore to keep track of the various data products.

Notes

These methods typically take a tag as input (i.e., something like “input”), but use the “aliased_tag” (i.e., something like “inform_pz_input”) when interacting with the DataStore.

In particular, the get_handle(), get_data() and input_iterator() will get the data from the DataStore under the aliased tag. E.g., if you call self.get_data(‘input’) for a Stage that has aliased “input” to “special_pz_input”, it will get the data associated to “special_pz_input” in the DataStore.

Similarly, add_handle() and set_data() will add the data to the DataStore under the aliased tag e.g., if you call self.set_data(‘input’) for a Stage that has aliased “input” to “special_pz_input”, it will store the data in the DataStore under the key “special_pz_input”.

And connect_input() will do the alias lookup both on the input and output. I.e., it is the same as calling self.set_data(inputTag, other.get_handle(outputTag, allow_missing=True), do_read=False)

__init__(args, **kwargs)

Constructor: Do RailStage specific initialization

Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

None

classmethod __new__(*args, **kwargs)

Running a rail stage

abstractmethod RailStage.run()

Run the stage and return the execution status.

Subclasses must implemented this method.

Note that subclasses of RailStage each implement a method that wraps rail.core.RailStage.run(), taking input data and properly attaching it to the stage, and wrapping the output data produced by the stage.

Building and configuring stages

classmethod RailStage.make_and_connect(**kwargs)

Make a stage and connects it to other stages

Notes

kwargs are used to set stage configuration, the should be key, value pairs, where the key is the parameter name and the value is value we want to assign

The ‘connections’ keyword is special, it is a dict[str, DataHandle] and should define the Input connections for this stage

Return type:

A stage

Parameters:

kwargs (Any)

classmethod RailStage.build(**kwargs)

Return an object that can be used to build a stage

Parameters:

kwargs (Any)

Return type:

RailStageBuild

RailStage.connect_input(other, inputTag=None, outputTag=None)

Connect another stage to this stage as an input

Parameters:

  • other (PipelineStage) – The stage whose output is being connected

  • inputTag (str | None) – Which input tag of this stage to connect to. None -> self.inputs[0]

  • outputTag (str | None) – Which output tag of the other stage to connect to. None -> other.outputs[0]

Returns:

The input handle for this stage

Return type:

DataHandle

Methods used by rail stage

RailStage.get_handle(tag, path=None, allow_missing=False)

Gets a DataHandle associated to a particular tag

Parameters:

  • tag (str) – The tag (from cls.inputs or cls.outputs) for this data

  • path (str | None) – The path to the data, only needed if we might need to read the data

  • allow_missing (bool) – If False this will raise a key error if the tag is not in the DataStore

Returns:

The handle that give access to the associated data

Return type:

DataHandle

RailStage.add_handle(tag, data=None, path=None)

Adds a DataHandle associated to a particular tag

Parameters:

  • tag (str) – The tag (from cls.inputs or cls.outputs) for this data

  • data (rail.core.data.DataLike) – If not None these data will be associated to the handle

  • path (str | None) – If not None, this will be the path used to read the data

Returns:

The handle that gives access to the associated data

Return type:

DataHandle

RailStage.input_iterator(tag, **kwargs)

Iterate the input assocated to a particular tag

Parameters:

  • tag (str) – The tag (from cls.inputs or cls.outputs) for this data

  • **kwargs (Any) – These will be passed to the Handle’s iterator method

Return type:

Fixme

RailStage.get_data(tag, allow_missing=True)

Gets the data associated to a particular tag

Notes

1. This gets the data via the DataHandle, and can and will read the data from disk if needed.

Parameters:

  • tag (str) – The tag (from cls.inputs or cls.outputs) for this data

  • allow_missing (bool) – If False this will raise a key error if the tag is not in the DataStore

Returns:

The data accesed by the handle assocated to the tag

Return type:

DataLike

RailStage.set_data(tag, data, path=None, do_read=True)

Sets the data associated to a particular tag

Notes

1. If data is a DataHandle and tag is one of the input tags, then this will add an alias between the two, i.e., it will set self.config.alias[tag] = data.tag. This allows the user to make connections between stages simply by passing DataHandles between them.

Parameters:

  • tag (str) – The tag (from cls.inputs or cls.outputs) for this data

  • data (rail.core.data.DataLike) – The data being set,

  • path (str | None) – Can be used to set the path for the data

  • do_read (bool) – If True, will read the data if it is not set

Returns:

The data accessed by the handle associated to the tag

Return type:

DataLike