Goldenspike - Interactive Version: an example of an end-to-end analysis using RAIL
Authors: Sam Schmidt, Eric Charles, Alex Malz, John Franklin Crenshaw, others…
Last run successfully: Feb 9, 2026
This notebook demonstrates how to use a the various RAIL Modules to draw synthetic samples of fluxes by color, apply physical effects to them, train photo-Z estimators on the samples, test and validate the preformance of those estimators, and to use the RAIL summarization modules to obtain n(z) estimates based on the p(z) estimates.
Note: If you’re interested in running this in pipeline mode, see
Goldenspike.ipynb
in the pipeline_examples/goldenspike_examples/ folder.
Creation
Note that in the parlance of the Creation Module, “degradation” is any post-processing that occurs to the “true” sample generated by the create Engine. This can include adding photometric errors, applying quality cuts, introducing systematic biases, etc.
In this notebook, we will draw both test and training samples from a RAIL Engine object. Then we will demonstrate how to use RAIL degraders to apply effects to those samples.
Training and Estimation
The RAIL Informer modules “train” or “inform” models used to estimate p(z) given band fluxes (and potentially other information).
The RAIL Estimation modules then use those same models to actually apply the model and extract the p(z) estimates.
p(z) Validation
The RAIL Validator module applies various metrics.
p(z) to n(z) Summarization
The RAIL Summarization modules convert per-galaxy p(z) posteriors to ensemble n(z) estimates.
import numpy as np
import rail.interactive as ri
import tables_io
from pzflow.examples import get_galaxy_data
Install FSPS with the following commands:
pip uninstall fsps
git clone --recursive https://github.com/dfm/python-fsps.git
cd python-fsps
python -m pip install .
export SPS_HOME=$(pwd)/src/fsps/libfsps
LEPHAREDIR is being set to the default cache directory which is being created at:
/home/runner/.cache/lephare/data
More than 1Gb may be written there.
LEPHAREWORK is being set to the default cache directory:
/home/runner/.cache/lephare/work
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.2.6 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/traitlets/config/application.py", line 1082, in launch_instance
app.start()
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 758, in start
self.io_loop.start()
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/utils.py", line 71, in preserve_context
return await f(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 621, in shell_main
await self.dispatch_shell(msg, subshell_id=subshell_id)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 478, in dispatch_shell
await result
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 372, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 834, in execute_request
reply_content = await reply_content
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 464, in do_execute
res = shell.run_cell(
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 663, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3077, in run_cell
result = self._run_cell(
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3132, in _run_cell
result = runner(coro)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 128, in _pseudo_sync_runner
coro.send(None)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3336, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3519, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3579, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_3917/1750766246.py", line 2, in <module>
import rail.interactive as ri
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/interactive/__init__.py", line 3, in <module>
from . import calib, creation, estimation, evaluation, tools
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/interactive/calib/__init__.py", line 3, in <module>
from rail.utils.interactive.initialize_utils import _initialize_interactive_module
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/utils/interactive/initialize_utils.py", line 17, in <module>
from rail.utils.interactive.base_utils import (
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/utils/interactive/base_utils.py", line 10, in <module>
rail.stages.import_and_attach_all(silent=True)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/stages/__init__.py", line 74, in import_and_attach_all
RailEnv.import_all_packages(silent=silent)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/core/introspection.py", line 541, in import_all_packages
_imported_module = importlib.import_module(pkg)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/som/__init__.py", line 1, in <module>
from rail.creation.degraders.specz_som import *
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/creation/degraders/specz_som.py", line 15, in <module>
from somoclu import Somoclu
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/somoclu/__init__.py", line 11, in <module>
from .train import Somoclu
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/somoclu/train.py", line 25, in <module>
from .somoclu_wrap import train as wrap_train
File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/somoclu/somoclu_wrap.py", line 11, in <module>
import _somoclu_wrap
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/numpy/core/_multiarray_umath.py:44, in __getattr__(attr_name)
39 # Also print the message (with traceback). This is because old versions
40 # of NumPy unfortunately set up the import to replace (and hide) the
41 # error. The traceback shouldn't be needed, but e.g. pytest plugins
42 # seem to swallow it and we should be failing anyway...
43 sys.stderr.write(msg + tb_msg)
---> 44 raise ImportError(msg)
46 ret = getattr(_multiarray_umath, attr_name, None)
47 if ret is None:
ImportError:
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.2.6 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Warning: the binary library cannot be imported. You cannot train maps, but you can load and analyze ones that you have already saved.
The problem occurs because either compilation failed when you installed Somoclu or a path is missing from the dependencies when you are trying to import it. Please refer to the documentation to see your options.
Here we need a few configuration parameters to deal with differences in data schema between existing PZ codes.
bands = ["u", "g", "r", "i", "z", "y"]
band_dict = {band: f"mag_{band}_lsst" for band in bands}
rename_dict = {f"mag_{band}_lsst_err": f"mag_err_{band}_lsst" for band in bands}
Get the data to use
catalog = get_galaxy_data().rename(band_dict, axis=1)
Train the Flow Engine
First we need to train the normalizing flow that will serve as the engine for the notebook.
flow_model = ri.creation.engines.flowEngine.flow_modeler(
input_data=catalog,
flow_seed=0,
phys_cols={"redshift": [0, 3]},
phot_cols={
"mag_u_lsst": [17, 35],
"mag_g_lsst": [16, 32],
"mag_r_lsst": [15, 30],
"mag_i_lsst": [15, 30],
"mag_z_lsst": [14, 29],
"mag_y_lsst": [14, 28],
},
calc_colors={"ref_column_name": "mag_i_lsst"},
)
Inserting handle into data store. input: None, FlowModeler
Training 30 epochs
Loss:
(0) 17.6137
(1) 0.8203
(2) -0.3311
(3) -1.2270
(4) -1.8016
(5) 0.6932
(6) -2.4607
(7) -2.7143
(8) -2.1091
(9) 2.9223
(10) -2.6319
(11) -2.6818
(12) -2.9049
(13) -2.3093
(14) -3.5844
(15) -2.7248
(16) -2.8104
(17) -3.1001
(18) -3.4940
(19) -2.7883
(20) -3.7015
(21) -3.0465
(22) -4.0176
(23) -2.8907
(24) -4.6285
(25) -4.5854
(26) -4.3568
(27) -1.9249
(28) -2.0749
(29) -4.0582
(30) -4.4706
Inserting handle into data store. model: inprogress_model.pkl, FlowModeler
ri.creation.engines.flowEngine.flow_modeler?
Make mock data
Now we will use the trained flow to create training and test data for the photo-z estimators.
For both the training and test data we will:
Use the Flow to produce some synthetic data
Use the LSSTErrorModel to add photometric errors
Use the FlowPosterior to estimate the redshift posteriors for the degraded sample
Use the ColumnMapper to rename the error columns so that they match the names in DC2.
Use the TableConverter to convert the data to a numpy dictionary, which will be stored in a hdf5 file with the same schema as the DC2 data
Training sample
For the training data we are going to apply a couple of extra degradation effects to the data beyond what we do to create test data, as the training data will have some spectroscopic incompleteness. This will allow us to see how the trained models perform with imperfect training data.
More details about the degraders are available in the
rail/examples/creation_examples/00_Quick_Start_in_Creation.ipynb
notebook.
train_data_orig = ri.creation.engines.flowEngine.flow_creator(
n_samples=150, model=flow_model["model"], seed=1235
)
Inserting handle into data store. model: <pzflow.flow.Flow object at 0x7f105daaba00>, FlowCreator
Inserting handle into data store. output: inprogress_output.pq, FlowCreator
train_data_errs = ri.creation.degraders.photometric_errors.lsst_error_model(
sample=train_data_orig["output"], seed=66, renameDict=band_dict, ndFlag=np.nan
)
Inserting handle into data store. input: None, LSSTErrorModel
Inserting handle into data store. output: inprogress_output.pq, LSSTErrorModel
train_data_inc = (
ri.creation.degraders.spectroscopic_degraders.inv_redshift_incompleteness(
sample=train_data_errs["output"], pivot_redshift=1.0
)
)
Inserting handle into data store. input: None, InvRedshiftIncompleteness
Inserting handle into data store. output: inprogress_output.pq, InvRedshiftIncompleteness
train_data_conf = ri.creation.degraders.spectroscopic_degraders.line_confusion(
sample=train_data_inc["output"],
true_wavelen=5007.0,
wrong_wavelen=3727.0,
frac_wrong=0.05,
seed=1337,
)
Inserting handle into data store. input: None, LineConfusion
Inserting handle into data store. output: inprogress_output.pq, LineConfusion
train_data_cut = ri.creation.degraders.quantityCut.quantity_cut(
sample=train_data_conf["output"], cuts={"mag_i_lsst": 25.0}
)
Inserting handle into data store. input: None, QuantityCut
Inserting handle into data store. output: inprogress_output.pq, QuantityCut
train_data_pq = ri.tools.table_tools.column_mapper(
data=train_data_cut["output"], columns=rename_dict
)
Inserting handle into data store. input: None, ColumnMapper
Inserting handle into data store. output: inprogress_output.pq, ColumnMapper
train_data = ri.tools.table_tools.table_converter(
data=train_data_pq["output"], output_format="numpyDict"
)
Inserting handle into data store. input: None, TableConverter
Inserting handle into data store. output: inprogress_output.hdf5, TableConverter
Let’s examine the quantities that we’ve generated, we’ll use the handy
tables_io package to temporarily write to a pandas dataframe for
quick writeout of the columns:
train_table = tables_io.convertObj(train_data["output"], tables_io.types.PD_DATAFRAME)
train_table.head()
| redshift | mag_u_lsst | mag_err_u_lsst | mag_g_lsst | mag_err_g_lsst | mag_r_lsst | mag_err_r_lsst | mag_i_lsst | mag_err_i_lsst | mag_z_lsst | mag_err_z_lsst | mag_y_lsst | mag_err_y_lsst | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.556576 | 24.570340 | 0.074358 | 23.965162 | 0.015322 | 22.818949 | 0.006747 | 22.119982 | 0.006393 | 21.829764 | 0.007734 | 21.551124 | 0.011598 |
| 1 | 0.156282 | 26.001843 | 0.253222 | 25.137768 | 0.041816 | 24.814243 | 0.027619 | 24.527944 | 0.034931 | 24.466716 | 0.063384 | 24.337835 | 0.126917 |
| 2 | 1.081121 | 25.727871 | 0.201793 | 25.779717 | 0.073801 | 25.517444 | 0.051412 | 24.930619 | 0.049914 | 24.468410 | 0.063479 | 24.269857 | 0.119645 |
| 3 | 1.034289 | 26.417749 | 0.353576 | 25.360323 | 0.050921 | 24.437050 | 0.019953 | 23.820884 | 0.018923 | 23.033678 | 0.018092 | 22.660453 | 0.028913 |
| 4 | 0.790806 | 25.795834 | 0.213581 | 25.483368 | 0.056784 | 25.133706 | 0.036582 | 24.671617 | 0.039666 | 24.459012 | 0.062952 | 24.278567 | 0.120554 |
You see that we’ve generated redshifts, ugrizy magnitudes, and magnitude errors with names that match those in the cosmoDC2_v1.1.4_image data.
Testing sample
For the test sample we will:
Use the Flow to produce some synthetic data
Use the LSSTErrorModel to smear the data
Use the FlowPosterior to estimate the redshift posteriors for the degraded sample
Use ColumnMapper to rename some of the columns to match DC2
Use the TableConverter to convert the data to a numpy dictionary, which will be stored in a hdf5 file with the same schema as the DC2 data
test_data_orig = ri.creation.engines.flowEngine.flow_creator(
model=flow_model["model"], n_samples=150, seed=1234
)
Inserting handle into data store. model: <pzflow.flow.Flow object at 0x7f105daaba00>, FlowCreator
Inserting handle into data store. output: inprogress_output.pq, FlowCreator
test_data_errs = ri.creation.degraders.photometric_errors.lsst_error_model(
sample=test_data_orig["output"], seed=58, renameDict=band_dict, ndFlag=np.nan
)
Inserting handle into data store. input: None, LSSTErrorModel
Inserting handle into data store. output: inprogress_output.pq, LSSTErrorModel
test_data_post = ri.creation.engines.flowEngine.flow_posterior(
input_data=test_data_errs["output"],
model=flow_model["model"],
column="redshift",
grid=np.linspace(0.0, 5.0, 21),
err_samples=None,
)
Inserting handle into data store. model: <pzflow.flow.Flow object at 0x7f105daaba00>, FlowPosterior
Inserting handle into data store. input: None, FlowPosterior
Inserting handle into data store. output: inprogress_output.hdf5, FlowPosterior
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/qp/parameterizations/interp/interp.py:187: UserWarning: The distributions at indices = [ 7 11 27] have an integral of 0.
warnings.warn(
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/qp/parameterizations/interp/interp.py:207: RuntimeWarning: invalid value encountered in divide
new_yvals = (self._yvals.T / self._ycumul[:, -1]).T
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/qp/parameterizations/interp/interp.py:208: RuntimeWarning: invalid value encountered in divide
self._ycumul = (self._ycumul.T / self._ycumul[:, -1]).T
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/qp/parameterizations/interp/interp.py:140: RuntimeWarning: There are non-finite values in the yvals for the following distributions: (array([ 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 27, 27, 27, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27]), array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 0, 1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]))
warnings.warn(
test_data_pq = ri.tools.table_tools.column_mapper(
data=test_data_errs["output"], columns=rename_dict, hdf5_groupname=""
)
Inserting handle into data store. input: None, ColumnMapper
Inserting handle into data store. output: inprogress_output.pq, ColumnMapper
test_data = ri.tools.table_tools.table_converter(
data=test_data_pq["output"], output_format="numpyDict"
)
Inserting handle into data store. input: None, TableConverter
Inserting handle into data store. output: inprogress_output.hdf5, TableConverter
test_table = tables_io.convertObj(test_data["output"], tables_io.types.PD_DATAFRAME)
test_table.head()
| redshift | mag_u_lsst | mag_err_u_lsst | mag_g_lsst | mag_err_g_lsst | mag_r_lsst | mag_err_r_lsst | mag_i_lsst | mag_err_i_lsst | mag_z_lsst | mag_err_z_lsst | mag_y_lsst | mag_err_y_lsst | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.576578 | 25.191498 | 0.127816 | 24.050870 | 0.016418 | 22.797245 | 0.006689 | 21.940036 | 0.006049 | 21.621931 | 0.007008 | 21.316804 | 0.009844 |
| 1 | 0.710664 | 26.317394 | 0.326651 | 26.273781 | 0.113845 | 25.762382 | 0.063895 | 25.127875 | 0.059468 | 24.908445 | 0.093619 | 24.845605 | 0.195905 |
| 2 | 0.960145 | 26.967345 | 0.535882 | 26.290216 | 0.115485 | 25.673743 | 0.059064 | 25.026222 | 0.054337 | 24.578019 | 0.069952 | 24.263974 | 0.119035 |
| 3 | 1.660866 | 28.426355 | 1.342814 | 29.028473 | 0.947352 | 27.925009 | 0.397267 | 26.878198 | 0.268532 | 25.914306 | 0.222001 | 27.029973 | 0.984762 |
| 4 | 0.505502 | 25.310711 | 0.141631 | 24.325771 | 0.020629 | 23.223199 | 0.008185 | 22.795467 | 0.008805 | 22.558163 | 0.012378 | 22.266115 | 0.020560 |
“Inform” some estimators
More details about the process of “informing” or “training” the models
used by the estimators is available in the
rail/examples/estimation_examples/00_Quick_Start_in_Estimation.ipynb
notebook.
We use “inform” rather than “train” to generically refer to the preprocessing of any prior information. For a machine learning estimator, that prior information is a training set, but it can also be an SED template library for a template-fitting or hybrid estimator.
inform_bpz = ri.estimation.algos.bpz_lite.bpz_lite_informer(
training_data=train_data["output"], nondetect_val=np.nan, hdf5_groupname=""
)
inform_knn = ri.estimation.algos.k_nearneigh.k_near_neigh_informer(
training_data=train_data["output"], nondetect_val=np.nan, hdf5_groupname=""
)
inform_fzboost = ri.estimation.algos.flexzboost.flex_z_boost_informer(
training_data=train_data["output"], nondetect_val=np.nan, hdf5_groupname=""
)
Inserting handle into data store. input: None, BPZliteInformer
Inserting handle into data store. model: inprogress_model.pkl, BPZliteInformer
Inserting handle into data store. input: None, KNearNeighInformer
split into 40 training and 14 validation samples
finding best fit sigma and NNeigh...
best fit values are sigma=0.075 and numneigh=3
Inserting handle into data store. model: inprogress_model.pkl, KNearNeighInformer
Inserting handle into data store. input: None, FlexZBoostInformer
stacking some data...
read in training data
fit the model...
finding best bump thresh...
finding best sharpen parameter...
Retraining with full training set...
Best bump = 0.05473684210526315, best sharpen = 1.1
Inserting handle into data store. model: inprogress_model.pkl, FlexZBoostInformer
Estimate photo-z posteriors
More detail on the specific estimators used here is available in the
rail/examples/estimation_examples/00_Quick_Start_in_Estimation.ipynb
notebook, but here is a very brief summary of the three estimators used
in this notebook:
BPZliteEstimator is a template-based photo-z code that outputs the
posterior estimated given likelihoods calculated using a template set
combined with a Bayesian prior. See Benitez (2000) for more details.
KNearNeighEstimator is a simple photo-z code that finds the K
nearest neighbor training galaxies in color/magnitude space and creates
a weighted (by distance) mixture model PDF based on the redshifts of
those K neighbors. FlexZBoostEstimator is a mature photo-z algorithm
that estimates a PDF for each galaxy via a conditional density estimate
using the training data. See Izbicki & Lee
(2017) for more details.
bpz_estimated = ri.estimation.algos.bpz_lite.bpz_lite_estimator(
input_data=test_data["output"],
model=inform_bpz["model"],
nondetect_val=np.nan,
hdf5_groupname="",
)
knn_estimated = ri.estimation.algos.k_nearneigh.k_near_neigh_estimator(
input_data=test_data["output"],
model=inform_knn["model"],
nondetect_val=np.nan,
hdf5_groupname="",
)
fzboost_estimated = ri.estimation.algos.flexzboost.flex_z_boost_estimator(
input_data=test_data["output"],
model=inform_fzboost["model"],
nondetect_val=np.nan,
hdf5_groupname="",
aliases={"input": "test_data", "output": "fzboost_estim"},
)
Inserting handle into data store. input: None, BPZliteEstimator
Inserting handle into data store. model: {'fo_arr': array([0.35, 0.5 ]), 'kt_arr': array([0.45 , 0.147]), 'zo_arr': array([0.431 , 0.39 , 0.0626]), 'km_arr': array([0.0913, 0.0636, 0.123 ]), 'a_arr': array([2.465, 1.806, 0.906]), 'mo': 20.0, 'nt_array': [1, 2, 5]}, BPZliteEstimator
Generating new AB file El_B2004a.DC2LSST_u.AB....
El_B2004a DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/El_B2004a.DC2LSST_u.AB
Generating new AB file El_B2004a.DC2LSST_g.AB....
El_B2004a DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/El_B2004a.DC2LSST_g.AB
Generating new AB file El_B2004a.DC2LSST_r.AB....
El_B2004a DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/El_B2004a.DC2LSST_r.AB
Generating new AB file El_B2004a.DC2LSST_i.AB....
El_B2004a DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/El_B2004a.DC2LSST_i.AB
Generating new AB file El_B2004a.DC2LSST_z.AB....
El_B2004a DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/El_B2004a.DC2LSST_z.AB
Generating new AB file El_B2004a.DC2LSST_y.AB....
El_B2004a DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/El_B2004a.DC2LSST_y.AB
Generating new AB file Sbc_B2004a.DC2LSST_u.AB....
Sbc_B2004a DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Sbc_B2004a.DC2LSST_u.AB
Generating new AB file Sbc_B2004a.DC2LSST_g.AB....
Sbc_B2004a DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Sbc_B2004a.DC2LSST_g.AB
Generating new AB file Sbc_B2004a.DC2LSST_r.AB....
Sbc_B2004a DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Sbc_B2004a.DC2LSST_r.AB
Generating new AB file Sbc_B2004a.DC2LSST_i.AB....
Sbc_B2004a DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Sbc_B2004a.DC2LSST_i.AB
Generating new AB file Sbc_B2004a.DC2LSST_z.AB....
Sbc_B2004a DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Sbc_B2004a.DC2LSST_z.AB
Generating new AB file Sbc_B2004a.DC2LSST_y.AB....
Sbc_B2004a DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Sbc_B2004a.DC2LSST_y.AB
Generating new AB file Scd_B2004a.DC2LSST_u.AB....
Scd_B2004a DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Scd_B2004a.DC2LSST_u.AB
Generating new AB file Scd_B2004a.DC2LSST_g.AB....
Scd_B2004a DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Scd_B2004a.DC2LSST_g.AB
Generating new AB file Scd_B2004a.DC2LSST_r.AB....
Scd_B2004a DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Scd_B2004a.DC2LSST_r.AB
Generating new AB file Scd_B2004a.DC2LSST_i.AB....
Scd_B2004a DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Scd_B2004a.DC2LSST_i.AB
Generating new AB file Scd_B2004a.DC2LSST_z.AB....
Scd_B2004a DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Scd_B2004a.DC2LSST_z.AB
Generating new AB file Scd_B2004a.DC2LSST_y.AB....
Scd_B2004a DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Scd_B2004a.DC2LSST_y.AB
Generating new AB file Im_B2004a.DC2LSST_u.AB....
Im_B2004a DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Im_B2004a.DC2LSST_u.AB
Generating new AB file Im_B2004a.DC2LSST_g.AB....
Im_B2004a DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Im_B2004a.DC2LSST_g.AB
Generating new AB file Im_B2004a.DC2LSST_r.AB....
Im_B2004a DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Im_B2004a.DC2LSST_r.AB
Generating new AB file Im_B2004a.DC2LSST_i.AB....
Im_B2004a DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Im_B2004a.DC2LSST_i.AB
Generating new AB file Im_B2004a.DC2LSST_z.AB....
Im_B2004a DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Im_B2004a.DC2LSST_z.AB
Generating new AB file Im_B2004a.DC2LSST_y.AB....
Im_B2004a DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/Im_B2004a.DC2LSST_y.AB
Generating new AB file SB3_B2004a.DC2LSST_u.AB....
SB3_B2004a DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB3_B2004a.DC2LSST_u.AB
Generating new AB file SB3_B2004a.DC2LSST_g.AB....
SB3_B2004a DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB3_B2004a.DC2LSST_g.AB
Generating new AB file SB3_B2004a.DC2LSST_r.AB....
SB3_B2004a DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB3_B2004a.DC2LSST_r.AB
Generating new AB file SB3_B2004a.DC2LSST_i.AB....
SB3_B2004a DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB3_B2004a.DC2LSST_i.AB
Generating new AB file SB3_B2004a.DC2LSST_z.AB....
SB3_B2004a DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB3_B2004a.DC2LSST_z.AB
Generating new AB file SB3_B2004a.DC2LSST_y.AB....
SB3_B2004a DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB3_B2004a.DC2LSST_y.AB
Generating new AB file SB2_B2004a.DC2LSST_u.AB....
SB2_B2004a DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB2_B2004a.DC2LSST_u.AB
Generating new AB file SB2_B2004a.DC2LSST_g.AB....
SB2_B2004a DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB2_B2004a.DC2LSST_g.AB
Generating new AB file SB2_B2004a.DC2LSST_r.AB....
SB2_B2004a DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB2_B2004a.DC2LSST_r.AB
Generating new AB file SB2_B2004a.DC2LSST_i.AB....
SB2_B2004a DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB2_B2004a.DC2LSST_i.AB
Generating new AB file SB2_B2004a.DC2LSST_z.AB....
SB2_B2004a DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB2_B2004a.DC2LSST_z.AB
Generating new AB file SB2_B2004a.DC2LSST_y.AB....
SB2_B2004a DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/SB2_B2004a.DC2LSST_y.AB
Generating new AB file ssp_25Myr_z008.DC2LSST_u.AB....
ssp_25Myr_z008 DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_25Myr_z008.DC2LSST_u.AB
Generating new AB file ssp_25Myr_z008.DC2LSST_g.AB....
ssp_25Myr_z008 DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_25Myr_z008.DC2LSST_g.AB
Generating new AB file ssp_25Myr_z008.DC2LSST_r.AB....
ssp_25Myr_z008 DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_25Myr_z008.DC2LSST_r.AB
Generating new AB file ssp_25Myr_z008.DC2LSST_i.AB....
ssp_25Myr_z008 DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_25Myr_z008.DC2LSST_i.AB
Generating new AB file ssp_25Myr_z008.DC2LSST_z.AB....
ssp_25Myr_z008 DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_25Myr_z008.DC2LSST_z.AB
Generating new AB file ssp_25Myr_z008.DC2LSST_y.AB....
ssp_25Myr_z008 DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_25Myr_z008.DC2LSST_y.AB
Generating new AB file ssp_5Myr_z008.DC2LSST_u.AB....
ssp_5Myr_z008 DC2LSST_u
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_5Myr_z008.DC2LSST_u.AB
Generating new AB file ssp_5Myr_z008.DC2LSST_g.AB....
ssp_5Myr_z008 DC2LSST_g
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_5Myr_z008.DC2LSST_g.AB
Generating new AB file ssp_5Myr_z008.DC2LSST_r.AB....
ssp_5Myr_z008 DC2LSST_r
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_5Myr_z008.DC2LSST_r.AB
Generating new AB file ssp_5Myr_z008.DC2LSST_i.AB....
ssp_5Myr_z008 DC2LSST_i
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_5Myr_z008.DC2LSST_i.AB
Generating new AB file ssp_5Myr_z008.DC2LSST_z.AB....
ssp_5Myr_z008 DC2LSST_z
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_5Myr_z008.DC2LSST_z.AB
Generating new AB file ssp_5Myr_z008.DC2LSST_y.AB....
ssp_5Myr_z008 DC2LSST_y
x_res[0] 3000.0
x_res[-1] 11500.0
Writing AB file /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/rail/examples_data/estimation_data/data/AB/ssp_5Myr_z008.DC2LSST_y.AB
Process 0 running estimator on chunk 0 - 150
Inserting handle into data store. output: inprogress_output.hdf5, BPZliteEstimator
Inserting handle into data store. input: None, KNearNeighEstimator
Inserting handle into data store. model: {'kdtree': <sklearn.neighbors._kd_tree.KDTree object at 0x55d7313913d0>, 'bestsig': np.float64(0.075), 'nneigh': 3, 'truezs': array([0.55657578, 0.15628164, 1.08112121, 1.03428864, 0.79080617,
0.5738259 , 1.33899605, 0.76665616, 1.22985588, 1.2558651 ,
0.23626536, 1.50679052, 0.40391368, 0.39398974, 0.67701137,
0.54411364, 0.58274353, 0.32774159, 0.66694409, 0.39556807,
1.66931748, 0.8531245 , 0.84963471, 0.45906514, 0.43978974,
0.77534676, 1.75164567, 0.90262765, 0.77140224, 0.67755717,
1.16743672, 0.79639375, 0.13033083, 0.55240422, 0.37811583,
0.77145851, 0.5949352 , 0.41755065, 1.3512429 , 0.94760352,
0.73229182, 0.71916658, 0.83944148, 0.09584825, 1.29376411,
1.02852273, 0.46523342, 1.23452401, 0.1253393 , 0.41696212,
1.09771186, 0.76246935, 0.39741305, 0.3066532 ]), 'only_colors': False}, KNearNeighEstimator
Process 0 running estimator on chunk 0 - 150
Process 0 estimating PZ PDF for rows 0 - 150
Inserting handle into data store. output: inprogress_output.hdf5, KNearNeighEstimator
Inserting handle into data store. test_data: None, FlexZBoostEstimator
Inserting handle into data store. model: <flexcode.core.FlexCodeModel object at 0x7f0fe3a95120>, FlexZBoostEstimator
Process 0 running estimator on chunk 0 - 150
Process 0 estimating PZ PDF for rows 0 - 150
Inserting handle into data store. fzboost_estim: inprogress_fzboost_estim.hdf5, FlexZBoostEstimator
Evaluate the estimates
Now we evaluate metrics on the estimates, separately for each estimator.
Each call to the Evaluator.evaluate will create a table with the
various performance metrics. We will store all of these tables in a
dictionary, keyed by the name of the estimator.
eval_dict = dict(bpz=bpz_estimated, fzboost=fzboost_estimated, knn=knn_estimated)
evaluator_stage_dict = dict(
metrics=["cdeloss", "pit", "brier"],
_random_state=None,
metric_config={
"brier": {"limits": (0, 3.1)},
"pit": {"tdigest_compression": 1000},
},
)
truth = test_data_orig
result_dict = {}
for key, val in eval_dict.items():
the_eval = ri.evaluation.dist_to_point_evaluator.dist_to_point_evaluator(
data=val["output"],
truth=truth["output"],
**evaluator_stage_dict,
hdf5_groupname="",
)
result_dict[key] = the_eval
WARNING:root:Input predictions do not sum to 1.
WARNING:root:Input predictions do not sum to 1.
WARNING:root:Input predictions do not sum to 1.
Inserting handle into data store. input: None, DistToPointEvaluator
Inserting handle into data store. truth: redshift mag_u_lsst mag_g_lsst mag_r_lsst mag_i_lsst mag_z_lsst 0 0.576578 25.116135 24.052393 22.783060 21.931549 21.616611
1 0.710664 26.642641 26.287365 25.734652 25.108639 24.887367
2 0.960145 26.522053 26.204634 25.669857 25.010715 24.509182
3 1.660866 29.665260 28.350712 27.750101 26.898108 26.247391
4 0.505502 25.260094 24.372253 23.218603 22.783365 22.563128
.. ... ... ... ... ... ...
145 0.301121 25.541651 24.488293 23.704102 23.466599 23.224829
146 0.904394 24.944120 24.650709 23.856140 22.962337 22.401966
147 2.184218 28.033718 27.158781 26.339005 26.024715 25.503304
148 0.752846 25.845242 24.625069 23.078152 21.886288 21.426828
149 1.166007 28.032230 27.204971 26.370100 25.790749 24.962671
mag_y_lsst
0 21.314606
1 24.780603
2 24.284958
3 25.794825
4 22.264570
.. ...
145 23.171350
146 22.187031
147 24.730652
148 21.228901
149 24.655422
[150 rows x 7 columns], DistToPointEvaluator
Requested metrics: ['cdeloss', 'pit', 'brier']
Inserting handle into data store. output: inprogress_output.hdf5, DistToPointEvaluator
Inserting handle into data store. summary: inprogress_summary.hdf5, DistToPointEvaluator
Inserting handle into data store. single_distribution_summary: inprogress_single_distribution_summary.hdf5, DistToPointEvaluator
Inserting handle into data store. input: None, DistToPointEvaluator
Inserting handle into data store. truth: redshift mag_u_lsst mag_g_lsst mag_r_lsst mag_i_lsst mag_z_lsst 0 0.576578 25.116135 24.052393 22.783060 21.931549 21.616611
1 0.710664 26.642641 26.287365 25.734652 25.108639 24.887367
2 0.960145 26.522053 26.204634 25.669857 25.010715 24.509182
3 1.660866 29.665260 28.350712 27.750101 26.898108 26.247391
4 0.505502 25.260094 24.372253 23.218603 22.783365 22.563128
.. ... ... ... ... ... ...
145 0.301121 25.541651 24.488293 23.704102 23.466599 23.224829
146 0.904394 24.944120 24.650709 23.856140 22.962337 22.401966
147 2.184218 28.033718 27.158781 26.339005 26.024715 25.503304
148 0.752846 25.845242 24.625069 23.078152 21.886288 21.426828
149 1.166007 28.032230 27.204971 26.370100 25.790749 24.962671
mag_y_lsst
0 21.314606
1 24.780603
2 24.284958
3 25.794825
4 22.264570
.. ...
145 23.171350
146 22.187031
147 24.730652
148 21.228901
149 24.655422
[150 rows x 7 columns], DistToPointEvaluator
Requested metrics: ['cdeloss', 'pit', 'brier']
Inserting handle into data store. output: inprogress_output.hdf5, DistToPointEvaluator
Inserting handle into data store. summary: inprogress_summary.hdf5, DistToPointEvaluator
Inserting handle into data store. single_distribution_summary: inprogress_single_distribution_summary.hdf5, DistToPointEvaluator
Inserting handle into data store. input: None, DistToPointEvaluator
Inserting handle into data store. truth: redshift mag_u_lsst mag_g_lsst mag_r_lsst mag_i_lsst mag_z_lsst 0 0.576578 25.116135 24.052393 22.783060 21.931549 21.616611
1 0.710664 26.642641 26.287365 25.734652 25.108639 24.887367
2 0.960145 26.522053 26.204634 25.669857 25.010715 24.509182
3 1.660866 29.665260 28.350712 27.750101 26.898108 26.247391
4 0.505502 25.260094 24.372253 23.218603 22.783365 22.563128
.. ... ... ... ... ... ...
145 0.301121 25.541651 24.488293 23.704102 23.466599 23.224829
146 0.904394 24.944120 24.650709 23.856140 22.962337 22.401966
147 2.184218 28.033718 27.158781 26.339005 26.024715 25.503304
148 0.752846 25.845242 24.625069 23.078152 21.886288 21.426828
149 1.166007 28.032230 27.204971 26.370100 25.790749 24.962671
mag_y_lsst
0 21.314606
1 24.780603
2 24.284958
3 25.794825
4 22.264570
.. ...
145 23.171350
146 22.187031
147 24.730652
148 21.228901
149 24.655422
[150 rows x 7 columns], DistToPointEvaluator
Requested metrics: ['cdeloss', 'pit', 'brier']
Inserting handle into data store. output: inprogress_output.hdf5, DistToPointEvaluator
Inserting handle into data store. summary: inprogress_summary.hdf5, DistToPointEvaluator
Inserting handle into data store. single_distribution_summary: inprogress_single_distribution_summary.hdf5, DistToPointEvaluator
result_dict
{'bpz': {'output': {},
'summary': {'cdeloss': array([-1.26942386]), 'brier': array([236.53229046])},
'single_distribution_summary': {'pit': Ensemble(the_class=quant,shape=(1, 97))}},
'fzboost': {'output': {},
'summary': {'cdeloss': array([-0.80740688]), 'brier': array([109.89122998])},
'single_distribution_summary': {'pit': Ensemble(the_class=quant,shape=(1, 95))}},
'knn': {'output': {},
'summary': {'cdeloss': array([-1.09976918]), 'brier': array([203.75672891])},
'single_distribution_summary': {'pit': Ensemble(the_class=quant,shape=(1, 90))}}}
The Pandas DataFrame output format conveniently makes human-readable
printouts of the metrics.
This next cell will convert everything to Pandas.
results_tables = {
key: tables_io.convertObj(val["summary"], tables_io.types.PD_DATAFRAME)
for key, val in result_dict.items()
}
results_tables["knn"]
| cdeloss | brier | |
|---|---|---|
| 0 | -1.099769 | 203.756729 |
results_tables["fzboost"]
| cdeloss | brier | |
|---|---|---|
| 0 | -0.807407 | 109.89123 |
results_tables["bpz"]
| cdeloss | brier | |
|---|---|---|
| 0 | -1.269424 | 236.53229 |
Summarize the per-galaxy redshift constraints to make population-level distributions
{introduce the summarizers}
First we make the stages, then execute them, then plot the output.
point_estimate_ens = ri.estimation.algos.point_est_hist.point_est_hist_summarizer(
input_data=eval_dict["bpz"]["output"]
)
naive_stack_ens = ri.estimation.algos.naive_stack.naive_stack_summarizer(
input_data=eval_dict["bpz"]["output"]
)
Inserting handle into data store. input: None, PointEstHistSummarizer
Process 0 running estimator on chunk 0 - 150
Inserting handle into data store. output: inprogress_output.hdf5, PointEstHistSummarizer
Inserting handle into data store. single_NZ: inprogress_single_NZ.hdf5, PointEstHistSummarizer
Inserting handle into data store. input: None, NaiveStackSummarizer
Process 0 running estimator on chunk 0 - 150
Inserting handle into data store. output: inprogress_output.hdf5, NaiveStackSummarizer
Inserting handle into data store. single_NZ: inprogress_single_NZ.hdf5, NaiveStackSummarizer
point_estimate_ens["output"].plot_native(xlim=(0, 3))
<Axes: xlabel='redshift', ylabel='p(z)'>
naive_stack_ens["output"].plot_native(xlim=(0, 3))
<Axes: xlabel='redshift', ylabel='p(z)'>
Note:
If you would like to convert an interactive workflow into a pipeline,
you will have to first convert the workflow into the make_stage()
and entrypoint_function() syntax used in old notebooks. Please take
a look at the [link to old Goldenspike here] notebook for how to perform
the same functions in this syntax, and then convert them to a pipeline.