rail.evaluation.evaluator module

Abstract base class defining an Evaluator

The key feature is that the evaluate method.

class rail.evaluation.evaluator.Evaluator

Bases: RailStage

Evaluate the performance of a photo-z estimator against reference point estimate

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • metrics ([list] default=[]) – The metrics you want to evaluate.

  • exclude_metrics ([list] default=[]) – List of metrics to exclude

  • metric_config ([dict] default={}) – configuration of individual_metrics

  • chunk_size ([int] default=10000) – Number of objects per chunk for parallel processing or to evalute per loop in single node processing

  • seed ([float] default=None) – Random seed value to use for reproducible results.

  • force_exact ([bool] default=False) – Force the exact calculation. This will not allow parallelization

  • output (Hdf5Handle (OUTPUT))

  • summary (Hdf5Handle (OUTPUT))

  • single_distribution_summary (QPDictHandle (OUTPUT))

__init__(args, **kwargs)

Constructor: Do RailStage specific initialization

Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

None

entrypoint_function: str | None = 'evaluate'
evaluate(data, truth, **kwargs)

Evaluate the performance of an estimator

This will attach the input data and truth to this Evaluator (for introspection and provenance tracking).

Then it will call the run() and finalize() methods, which need to be implemented by the sub-classes.

The run() method will need to register the data that it creates to this Estimator by using self.add_data(‘output’, output_data).

Parameters:

  • data (qp.Ensemble) – The sample to evaluate

  • truth (Any) – Table with the truth information

Returns:

The evaluation metrics

Return type:

dict[str, DataHandle]

finalize()

Finalize the stage, moving all its outputs to their final locations.

Return type:

None

inputs: list[tuple[str, type[DataHandle]]] = []
metric_base_class: type[BaseMetric] | None = None
name = 'Evaluator'
outputs = [('output', <class 'rail.core.data.Hdf5Handle'>), ('summary', <class 'rail.core.data.Hdf5Handle'>), ('single_distribution_summary', <class 'rail.core.data.QPDictHandle'>)]
run()

Run the stage and return the execution status.

Subclasses must implemented this method.

Return type:

None

run_single_node()
Return type:

None

class rail.evaluation.evaluator.OldEvaluator

Bases: RailStage

Evaluate the performance of a photo-Z estimator

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • zmin ([float] default=0.0) – The minimum redshift of the z grid or sample

  • zmax ([float] default=3.0) – The maximum redshift of the z grid or sample

  • nzbins ([int] default=301) – The number of gridpoints in the z grid

  • pit_metrics ([str] default=all) – PIT-based metrics to include

  • point_metrics ([str] default=all) – Point-estimate metrics to include

  • hdf5_groupname ([str] default=) – name of hdf5 group for data, if None, then set to ‘’

  • do_cde ([bool] default=True) – Evaluate CDE Metric

  • redshift_col ([str] default=redshift) – name of redshift column

  • input (QPHandle (INPUT))

  • truth (Hdf5Handle (INPUT))

  • output (Hdf5Handle (OUTPUT))

entrypoint_function: str | None = 'evaluate'
evaluate(data, truth, **kwargs)

Evaluate the performance of an estimator

This will attach the input data and truth to this Evaluator (for introspection and provenance tracking). Then it will call the run() and finalize() methods, which need to be implemented by the sub-classes. The run() method will need to register the data that it creates to this Estimator by using self.add_data(‘output’, output_data).

Parameters:

  • data (qp.Ensemble) – The sample to evaluate

  • truth (Any) – Table with the truth information

Returns:

The evaluation metrics

Return type:

Hdf5Handle

inputs = [('input', <class 'rail.core.data.QPHandle'>), ('truth', <class 'rail.core.data.Hdf5Handle'>)]
interactive_function: str | None = 'old_evaluator'
name = 'OldEvaluator'
outputs = [('output', <class 'rail.core.data.Hdf5Handle'>)]
run()

Run method Evaluate all the metrics and put them into a table .. rubric:: Notes

Get the input data from the data store under this stages ‘input’ tag Get the truth data from the data store under this stages ‘truth’ tag Puts the data into the data store under this stages ‘output’ tag

Return type:

None