Evaluation

Evaluation is a type of RAIL stage which compares estimated photometric redshifts to known true values, in order to assess the performance of the estimation algorithm.

../../_images/evaluation.png

Evaluators

Evaluators evaluate the performance of a photo-z estimator against reference point estimate.

Distribution to Distribution

RAIL Package: https://github.com/LSSTDESC/rail_base

Evaluate the performance of a photo-z estimator against reference PDFs.

class rail.evaluation.dist_to_dist_evaluator.DistToDistEvaluator

Evaluate the performance of a photo-z estimator against reference PDFs

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • metrics ([list] default=[]) – The metrics you want to evaluate.

  • exclude_metrics ([list] default=[]) – List of metrics to exclude

  • metric_config ([dict] default={}) – configuration of individual_metrics

  • chunk_size ([int] default=10000) – Number of objects per chunk for parallel processing or to evalute per loop in single node processing

  • seed ([float] default=None) – Random seed value to use for reproducible results.

  • force_exact ([bool] default=False) – Force the exact calculation. This will not allow parallelization

  • metric_integration_limits ([list] default=[0.0, 3.0]) – The default end points for calculating metrics on a grid.

  • dx ([float] default=0.01) – The default step size when calculating metrics on a grid.

  • n_samples ([int] default=100) – The number of random samples to select for certain metrics.

  • input (QPHandle (INPUT))

  • truth (QPHandle (INPUT))

  • output (Hdf5Handle (OUTPUT))

  • summary (Hdf5Handle (OUTPUT))

  • single_distribution_summary (QPDictHandle (OUTPUT))

__init__(args, **kwargs)

Constructor: Do RailStage specific initialization

Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

None

classmethod __new__(*args, **kwargs)

Distribution to Point

RAIL Package: https://github.com/LSSTDESC/rail_base

Evaluate the performance of a photo-z estimator against reference point estimate.

class rail.evaluation.dist_to_point_evaluator.DistToPointEvaluator

Evaluate the performance of a photo-z estimator against reference point estimate

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • metrics ([list] default=[]) – The metrics you want to evaluate.

  • exclude_metrics ([list] default=[]) – List of metrics to exclude

  • metric_config ([dict] default={}) – configuration of individual_metrics

  • chunk_size ([int] default=10000) – Number of objects per chunk for parallel processing or to evalute per loop in single node processing

  • seed ([float] default=None) – Random seed value to use for reproducible results.

  • force_exact ([bool] default=False) – Force the exact calculation. This will not allow parallelization

  • metric_integration_limits ([list] default=[0.0, 3.0]) – The default end points for calculating metrics on a grid.

  • dx ([float] default=0.01) – The default step size when calculating metrics on a grid.

  • quantile_grid ([list] (default=[...])) – The quantile value grid on which to evaluate the CDF values. (0, 1)

  • x_grid ([list] (default=[...])) – The x-value grid at which to evaluate the pdf values.

  • hdf5_groupname ([str] default=photometry) – name of hdf5 group for data, if None, then set to ‘’

  • reference_dictionary_key ([str] default=redshift) – The key in the truth dictionary where the redshift data is stored.

  • input (QPHandle (INPUT))

  • truth (TableHandle (INPUT))

  • output (Hdf5Handle (OUTPUT))

  • summary (Hdf5Handle (OUTPUT))

  • single_distribution_summary (QPDictHandle (OUTPUT))

__init__(args, **kwargs)

Constructor: Do RailStage specific initialization

Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

None

classmethod __new__(*args, **kwargs)

Point to Point

RAIL Package: https://github.com/LSSTDESC/rail_base

Evaluate the performance of a photo-z estimator against reference point estimate.

class rail.evaluation.point_to_point_evaluator.PointToPointEvaluator

Evaluate the performance of a photo-z estimator against reference point estimate

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • metrics ([list] default=[]) – The metrics you want to evaluate.

  • exclude_metrics ([list] default=[]) – List of metrics to exclude

  • metric_config ([dict] default={}) – configuration of individual_metrics

  • chunk_size ([int] default=10000) – Number of objects per chunk for parallel processing or to evalute per loop in single node processing

  • seed ([float] default=None) – Random seed value to use for reproducible results.

  • force_exact ([bool] default=False) – Force the exact calculation. This will not allow parallelization

  • hdf5_groupname ([str] default=photometry) – name of hdf5 group for data, if None, then set to ‘’

  • reference_dictionary_key ([str] default=redshift) – The key in the truth dictionary where the redshift data is stored.

  • point_estimate_key ([str] default=zmode) – The key in the point estimate table.

  • input (QPHandle (INPUT))

  • truth (TableHandle (INPUT))

  • output (Hdf5Handle (OUTPUT))

  • summary (Hdf5Handle (OUTPUT))

  • single_distribution_summary (QPDictHandle (OUTPUT))

__init__(args, **kwargs)

Constructor: Do RailStage specific initialization

Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

None

classmethod __new__(*args, **kwargs)

Single

RAIL Package: https://github.com/LSSTDESC/rail_base

class rail.evaluation.single_evaluator.SingleEvaluator

Evaluate the performance of a photo-Z estimator

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • metrics ([list] default=[]) – The metrics you want to evaluate.

  • exclude_metrics ([list] default=[]) – List of metrics to exclude

  • metric_config ([dict] default={}) – configuration of individual_metrics

  • chunk_size ([int] default=10000) – Number of objects per chunk for parallel processing or to evalute per loop in single node processing

  • seed ([float] default=None) – Random seed value to use for reproducible results.

  • force_exact ([bool] default=False) – Force the exact calculation. This will not allow parallelization

  • point_estimates ([list] default=[]) – List of point estimates to use

  • truth_point_estimates ([list] default=[]) – List of true point values to use

  • hdf5_groupname ([str] default=photometry) – name of hdf5 group for data, if None, then set to ‘’

  • input (QPOrTableHandle (INPUT))

  • truth (QPOrTableHandle (INPUT))

  • output (Hdf5Handle (OUTPUT))

  • summary (Hdf5Handle (OUTPUT))

  • single_distribution_summary (QPDictHandle (OUTPUT))

__init__(args, **kwargs)

Initialize Evaluator

Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

None

classmethod __new__(*args, **kwargs)