rail.estimation.algos.minisom_som module

class rail.estimation.algos.minisom_som.MiniSOMInformer

Bases: CatInformer

Summarizer that uses a SOM to construct a weighted sum of spec-z objects in the same SOM cell as each photometric galaxy in order to estimate the overall N(z). This is very related to the NZDir estimator, though that estimator actually reverses this process and looks for photometric neighbors around each spectroscopic galaxy, which can lead to problems if there are photometric galaxies with no nearby spec-z objects (NZDir is not aware that such objects exist and thus can hid biases). Part of the SimpeSOM estimator will be a check for cells which contain photometric objects but do not contain any corresponding training/spec-z objects, those unmatched objects will be flagged for possible removal from the input sample. The inform stage will simply construct a 2D grid SOM using minisom from a large sample of input photometric data and save this as an output. This may be a computationally intensive stage, though it will hopefully be run once and used by the estimate/summarize stage many times without needing to be re-run.

We can make the SOM either with all colors, or one magnitude and N colors, or an arbitrary set of columns. The code includes a flag column_usage to set usage, If set to “colors” it will take the difference of each adjacen pair of columns in bands as the colors. If set to magandcolors it will use these colors plus one magnitude as specified by ref_band. If set to columns then it will take as inputs all of the columns specified by bands (they can be magnitudes, colors, or any other input specified by the user). NOTE: any custom bands parameters must have an accompanying nondetect_val dictionary that will replace nondetections with the nondetect_val values!

This will make a pickle file containing the minisom SOM object that will be used by the estimation/summarization stage

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • hdf5_groupname (str] (default=photometry))

  • nondetect_val (float] (default=99.0))

  • mag_limits (dict] (default={'mag_u_lsst': 27.79, 'mag_g_lsst': 29.04, 'mag_r_lsst': 29.06, 'mag_i_lsst': 28.62, 'mag_z_lsst': 27.98, 'mag_y_lsst': 27.05}))

  • bands (list] (default=['mag_u_lsst', 'mag_g_lsst', 'mag_r_lsst', 'mag_i_lsst', 'mag_z_lsst', 'mag_y_lsst']))

  • ref_band (str] (default=mag_i_lsst))

  • column_usage ([str] default=magandcolors) – switch for how SOM uses columns, valid values are ‘colors’, ‘magandcolors’, and ‘columns’

  • seed ([int] default=0) – Random number seed

  • m_dim ([int] default=31) – number of cells in SOM y dimension

  • n_dim ([int] default=31) – number of cells in SOM x dimension

  • som_sigma ([float] default=1.5) – sigma param in SOM training

  • som_learning_rate ([float] default=0.5) – SOM learning rate

  • som_iterations ([int] default=10000) – number of iterations in SOM training

  • input (TableHandle (INPUT))

  • model (ModelHandle (OUTPUT))

__init__(args, **kwargs)

Constructor: Do Informer specific initialization

entrypoint_function: str | None = 'inform'
interactive_function: str | None = 'mini_som_informer'
name = 'MiniSOMInformer'
run()

Build a SOM from photometric data NOT spectroscopic data!

class rail.estimation.algos.minisom_som.MiniSOMSummarizer

Bases: SZPZSummarizer

Quick implementation of a SOM-based summarizer that constructs and N(z) estimate via a weighted sum of the empirical N(z) consisting of the normalized histogram of spec-z values contained in the same SOM cell as each photometric galaxy. There are some general guidelines to choosing the geometry and number of total cells in the SOM. This paper: http://www.giscience2010.org/pdfs/paper_230.pdf recommends 5*sqrt(num rows * num data columns) as a rough guideline. Some authors state that a SOM with one dimension roughly twice as long as the other are better, while others find that square SOMs with equal X and Y dimensions are best, the user can set the dimensions using the n_dim and m_dim parameters. For more discussion on SOMs and photo-z calibration, see the KiDS paper on the topic: http://arxiv.org/abs/1909.09632 particularly the appendices. Note that several parameters are stored in the model file, e.g. the columns used. This ensures that the same columns used in constructing the SOM are used when finding the winning SOM cell with the test data. Two additional files are also written out: cellid_output outputs the ‘winning’ SOM cell for each photometric galaxy, in both raveled and 2D SOM cell coordinates. If the objectID or galaxy_id is present they will also be included in this file, if not the coordinates will be written in the same order in which the data is read in. uncovered_cell_file outputs the raveled cell IDs of cells that contain photometric galaxies but no corresponding spectroscopic objects, these objects should be removed from the sample as they cannot be accounted for properly in the summarizer. Some iteration on data cuts may be necessary to remove/mitigate these ‘uncovered’ objects.

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • chunk_size ([int] default=10000) – Number of objects per chunk for parallel processing or to evalute per loop in single node processing

  • zmin (float] (default=0.0))

  • zmax (float] (default=3.0))

  • nzbins (int] (default=301))

  • nondetect_val (float] (default=99.0))

  • mag_limits (dict] (default={'mag_u_lsst': 27.79, 'mag_g_lsst': 29.04, 'mag_r_lsst': 29.06, 'mag_i_lsst': 28.62, 'mag_z_lsst': 27.98, 'mag_y_lsst': 27.05}))

  • hdf5_groupname (str] (default=photometry))

  • redshift_col (str] (default=redshift))

  • objid_name ([str] default=) – name of ID column, if present will be written to cellid_output

  • spec_groupname ([str] default=photometry) – name of hdf5 group for spec data, if None, then set to ‘’

  • seed ([int] default=12345) – random seed

  • phot_weightcol ([str] default=) – name of photometry weight, if present

  • spec_weightcol ([str] default=) – name of specz weight col, if present

  • n_samples ([int] default=20) – number of bootstrap samples to generate

  • input (TableHandle (INPUT))

  • spec_input (TableHandle (INPUT))

  • model (ModelHandle (INPUT))

  • output (QPHandle (OUTPUT))

  • single_NZ (QPHandle (OUTPUT))

  • cellid_output (TableHandle (OUTPUT))

  • uncovered_cell_file (TableHandle (OUTPUT))

__init__(args, **kwargs)

Initialize Estimator that can sample galaxy data.

entrypoint_function: str | None = 'summarize'
interactive_function: str | None = 'mini_som_summarizer'
name = 'MiniSOMSummarizer'
outputs = [('output', <class 'rail.core.data.QPHandle'>), ('single_NZ', <class 'rail.core.data.QPHandle'>), ('cellid_output', <class 'rail.core.data.TableHandle'>), ('uncovered_cell_file', <class 'rail.core.data.TableHandle'>)]
run()

Run the stage and return the execution status.

Subclasses must implemented this method.