rail.estimation.algos.somoclu_som module

class rail.estimation.algos.somoclu_som.SOMocluInformer

Bases: CatInformer

Summarizer that uses a SOM to construct a weighted sum of spec-z objects in the same SOM cell as each photometric galaxy in order to estimate the overall N(z). This is very related to the NZDir estimator, though that estimator actually reverses this process and looks for photometric neighbors around each spectroscopic galaxy, which can lead to problems if there are photometric galaxies with no nearby spec-z objects (NZDir is not aware that such objects exist and thus can hid biases).

We apply somoclu package (https://somoclu.readthedocs.io/) to train the SOM.

Part of the SOM estimator will be a check for cells which contain photometric objects but do not contain any corresponding training/spec-z objects, those unmatched objects will be flagged for possible removal from the input sample. The inform stage will simply construct a 2D grid SOM using somoclu from a large sample of input photometric data and save this as an output. This may be a computationally intensive stage, though it will hopefully be run once and used by the estimate/summarize stage many times without needing to be re-run.

We can make the SOM either with all colors, or one magnitude and N colors, or an arbitrary set of columns. The code includes a flag column_usage to set usage, If set to “colors” it will take the difference of each adjacen pair of columns in bands as the colors. If set to magandcolors it will use these colors plus one magnitude as specified by ref_band. If set to columns then it will take as inputs all of the columns specified by bands (they can be magnitudes, colors, or any other input specified by the user). NOTE: any custom bands parameters must have an accompanying nondetect_val dictionary that will replace nondetections with the nondetect_val values!

This creates a pickle file containing the somoclu SOM object that will be used by the estimation/summarization stage

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • hdf5_groupname (str] (default=photometry))

  • nondetect_val (float] (default=99.0))

  • mag_limits (dict] (default={'mag_u_lsst': 27.79, 'mag_g_lsst': 29.04, 'mag_r_lsst': 29.06, 'mag_i_lsst': 28.62, 'mag_z_lsst': 27.98, 'mag_y_lsst': 27.05}))

  • bands (list] (default=['mag_u_lsst', 'mag_g_lsst', 'mag_r_lsst', 'mag_i_lsst', 'mag_z_lsst', 'mag_y_lsst']))

  • err_bands (list] (default=['mag_err_u_lsst', 'mag_err_g_lsst', 'mag_err_r_lsst', 'mag_err_i_lsst', 'mag_err_z_lsst', 'mag_err_y_lsst']))

  • ref_band (str] (default=mag_i_lsst))

  • redshift_col (str] (default=redshift))

  • column_usage ([str] default=magandcolors) – switch for how SOM uses columns, valid values are ‘colors’,’magandcolors’, and ‘mags’

  • seed ([int] default=0) – Random number seed

  • n_rows ([int] default=31) – number of cells in SOM y dimension

  • n_columns ([int] default=31) – number of cells in SOM x dimension

  • grid_type ([str] default=rectangular) – Optional parameter to specify the grid form of the nodes:* ‘rectangular’: rectangular neurons (default)* ‘hexagonal’: hexagonal neurons

  • n_epochs ([int] default=10) – number of training epochs.

  • initialization ([str] default=pca) – method of initializing the SOM:* ‘pca’: principal componant analysis (default)* ‘random’ randomly initialize the SOM

  • maptype ([str] default=planar) – Optional parameter to specify the map topology:* ‘planar’: Planar map (default)* ‘toroid’: Toroid map

  • std_coeff ([float] default=1.5) – Optional parameter to set the coefficient in the Gaussianneighborhood function exp(-||x-y||^2/(2*(coeff*radius)^2))Default: 1.5

  • som_learning_rate ([float] default=0.5) – Initial SOM learning rate (scale0 param in Somoclu)

  • input (TableHandle (INPUT))

  • model (ModelHandle (OUTPUT))

__init__(args, **kwargs)

Constructor: Do Informer specific initialization

entrypoint_function: str | None = 'inform'
interactive_function: str | None = 'somoclu_informer'
name = 'SOMocluInformer'
run()

Build a SOM from photometric data NOT spectroscopic data!

class rail.estimation.algos.somoclu_som.SOMocluSummarizer

Bases: SZPZSummarizer

Quick implementation of a SOM-based summarizer. It will group a pre-trained SOM into hierarchical clusters and assign a galaxy sample into SOM cells and clusters. Then it constructs an N(z) estimation via a weighted sum of the empirical N(z) consisting of the normalized histogram of spec-z values contained in the same SOM cluster as each photometric galaxy. There are some general guidelines to choosing the geometry and number of total cells in the SOM. This paper: http://www.giscience2010.org/pdfs/paper_230.pdf recommends 5*sqrt(num rows * num data columns) as a rough guideline. Some authors state that a SOM with one dimension roughly twice as long as the other are better, while others find that square SOMs with equal X and Y dimensions are best, the user can set the dimensions using the n_columns and n_rows parameters. For more discussion on SOMs and photo-z calibration, see the KiDS paper on the topic: http://arxiv.org/abs/1909.09632 particularly the appendices. Note that several parameters are stored in the model file, e.g. the columns used. This ensures that the same columns used in constructing the SOM are used when finding the winning SOM cell with the test data. Two additional files are also written out: cellid_output outputs the ‘winning’ SOM cell for each photometric galaxy, in both raveled and 2D SOM cell coordinates. If the objectID or galaxy_id is present they will also be included in this file, if not the coordinates will be written in the same order in which the data is read in. uncovered_cell_file outputs the raveled cell IDs of cells that contain photometric galaxies but no corresponding spectroscopic objects, these objects should be removed from the sample as they cannot be accounted for properly in the summarizer. Some iteration on data cuts may be necessary to remove/mitigate these ‘uncovered’ objects.

Parameters:

  • output_mode ([str] default=default) – What to do with the outputs. The options are ‘default’, where outputs will be written to files and some returned, and ‘return’, where outputs will only be returned and not written.

  • chunk_size ([int] default=10000) – Number of objects per chunk for parallel processing or to evalute per loop in single node processing

  • zmin (float] (default=0.0))

  • zmax (float] (default=3.0))

  • nzbins (int] (default=301))

  • nondetect_val (float] (default=99.0))

  • mag_limits (dict] (default={'mag_u_lsst': 27.79, 'mag_g_lsst': 29.04, 'mag_r_lsst': 29.06, 'mag_i_lsst': 28.62, 'mag_z_lsst': 27.98, 'mag_y_lsst': 27.05}))

  • hdf5_groupname (str] (default=photometry))

  • redshift_col (str] (default=redshift))

  • spec_groupname ([str] default=photometry) – name of hdf5 group for spec data, if None, then set to ‘’

  • n_clusters ([int] default=-1) – The number of hierarchical clusters of SOM cells. If not provided, the SOM cells will not be clustered.

  • objid_name ([str] default=) – name of ID column, if present will be written to cellid_output

  • seed ([int] default=12345) – random seed

  • phot_weightcol ([str] default=) – name of photometry weight, if present

  • spec_weightcol ([str] default=) – name of specz weight col, if present

  • som_split_size ([int] default=200) – the size of data chunks when calculating the distances between the codebook and data

  • n_samples ([int] default=20) – number of bootstrap samples to generate

  • useful_clusters ([list] default=[]) – the cluster indices that are used for calibration. If not given, then all the clusters containing spec sample are used.

  • input (TableHandle (INPUT))

  • spec_input (TableHandle (INPUT))

  • model (ModelHandle (INPUT))

  • output (QPHandle (OUTPUT))

  • single_NZ (QPHandle (OUTPUT))

  • cellid_output (Hdf5Handle (OUTPUT))

  • uncovered_cluster_file (TableHandle (OUTPUT))

__init__(args, **kwargs)

Initialize Estimator that can sample galaxy data.

entrypoint_function: str | None = 'summarize'
get_som_coordinates(data, weight_col)

Find the bmus coordinate of each item in the data.

interactive_function: str | None = 'somoclu_summarizer'
name = 'SOMocluSummarizer'
outputs = [('output', <class 'rail.core.data.QPHandle'>), ('single_NZ', <class 'rail.core.data.QPHandle'>), ('cellid_output', <class 'rail.core.data.Hdf5Handle'>), ('uncovered_cluster_file', <class 'rail.core.data.TableHandle'>)]
replace_non_detections(data)

Replace non-detected data with magnitude limits.

run()

Run the stage and return the execution status.

Subclasses must implemented this method.

set_weight_column(data, weight_col)

Assign weight vecs if present, else set all to 1.0. weight_col: column name of weights.

rail.estimation.algos.somoclu_som.get_bmus(som, data, split=200)

This function gets the “best matching unit (bmu)” of a given data on a pre-trained SOM. It works by multiprocessing chunks of the data. Input: som: a pre-trained Somoclu object; data: np.ndarray of the data vector. split: an integer specifying the size of data chunks when calculating the distances between the codebook and data;

rail.estimation.algos.somoclu_som.plot_som(ax, som_map, grid_type='rectangular', colormap=<matplotlib.colors.ListedColormap object>, cbar_name=None, vmin=None, vmax=None)

This function plots the pre-trained SOM. Input: ax: the axis to be plotted on. som_map: a 2-D array contains the value in a pre-trained SOM. The value can be the number of sources in each cell; or the mean feature in every cell. grid_type: string, either ‘rectangular’ or ‘hexagonal’. colormap: the colormap to show the values. default: cm.viridis. cbar_name: the label on the color bar.