Running FastSpecFit 

This page covers running FastSpecFit interactively on individual objects and at scale across large datasets. For reading and working with existing output catalogs, see Working with FastSpecFit Data. For worked examples, see the Tutorials page.

Fitting Individual Objects 

Note

The examples in this section and in Fitting at Scale assume that FastSpecFit has been successfully installed and set up.

fastspec jointly models the DESI spectrophotometry and broadband photometry for one or more objects, taking one or more Redrock redshift catalogs as input. fastphot models only the broadband photometry and is significantly faster. Both tools write results to a multi-extension FITS file described in the data model pages.

Click to view the fastspec help message.

fastspec --help
usage: fastspec [-h] -o OUTFILE [--mp MP] [-n NTARGETS] [--firsttarget FIRSTTARGET] [--targetids TARGETIDS] [--input-redshifts INPUT_REDSHIFTS] [--input-seeds INPUT_SEEDS]
                [--seed SEED] [--nmonte NMONTE] [--vdisp-nominal VDISP_NOMINAL] [--vdisp-bounds VDISP_BOUNDS VDISP_BOUNDS] [--zmin ZMIN] [--no-broadlinefit]
                [--ignore-photometry] [--ignore-quasarnet] [--constrain-age] [--no-smooth-continuum] [--imf IMF] [--templateversion TEMPLATEVERSION] [--templates TEMPLATES]
                [--redrockfile-prefix REDROCKFILE_PREFIX] [--specfile-prefix SPECFILE_PREFIX] [--qnfile-prefix QNFILE_PREFIX] [--mapdir MAPDIR] [--fphotodir FPHOTODIR]
                [--fphotofile FPHOTOFILE] [--emlinesfile EMLINESFILE] [--redux_dir REDUX_DIR] [--specproddir SPECPRODDIR] [--uncertainty-floor UNCERTAINTY_FLOOR]
                [--minsnr-balmer-broad MINSNR_BALMER_BROAD] [--debug-plots] [--verbose]
                redrockfiles [redrockfiles ...]

positional arguments:
  redrockfiles          Full path to input redrock file(s).

options:
  -h, --help            show this help message and exit
  -o OUTFILE, --outfile OUTFILE
                        Full path to output filename (required). (default: None)
  --mp MP               Number of multiprocessing threads per MPI rank. (default: 1)
  -n NTARGETS, --ntargets NTARGETS
                        Number of targets to process in each file. (default: None)
  --firsttarget FIRSTTARGET
                        Index of first object to to process in each file, zero-indexed. (default: 0)
  --targetids TARGETIDS
                        Comma-separated list of TARGETIDs to process. (default: None)
  --input-redshifts INPUT_REDSHIFTS
                        Comma-separated list of input redshifts corresponding to the (required) --targetids input. (default: None)
  --input-seeds INPUT_SEEDS
                        Comma-separated list of input random-number seeds corresponding to the (required) --targetids input. (default: None)
  --seed SEED           Random seed for Monte Carlo reproducibility; ignored if --input-seeds is passed. (default: 1)
  --nmonte NMONTE       Number of Monte Carlo realizations. (default: 50)
  --vdisp-nominal VDISP_NOMINAL
                        Nominal (default) velocity dispersion in km/s. (default: 250.0)
  --vdisp-bounds VDISP_BOUNDS VDISP_BOUNDS
                        Nominal (default) velocity dispersion in km/s. (default: (75.0, 500.0))
  --zmin ZMIN           Override the default minimum redshift required for modeling. (default: None)
  --no-broadlinefit     Do not model broad Balmer and helium line-emission. (default: True)
  --ignore-photometry   Ignore the broadband photometry during model fitting. (default: False)
  --ignore-quasarnet    Do not use QuasarNet to improve QSO redshifts. (default: True)
  --constrain-age       Constrain the age of the SED. (default: False)
  --no-smooth-continuum
                        Do not fit the smooth continuum. (default: False)
  --imf IMF             Initial mass function. (default: chabrier)
  --templateversion TEMPLATEVERSION
                        Template version number. (default: 2.0.0)
  --templates TEMPLATES
                        Optional full path and filename to the templates. (default: None)
  --redrockfile-prefix REDROCKFILE_PREFIX
                        Prefix of the input Redrock file name(s). (default: redrock-)
  --specfile-prefix SPECFILE_PREFIX
                        Prefix of the spectral file(s). (default: coadd-)
  --qnfile-prefix QNFILE_PREFIX
                        Prefix of the QuasarNet afterburner file(s). (default: qso_qn-)
  --mapdir MAPDIR       Optional directory name for the dust maps. (default: None)
  --fphotodir FPHOTODIR
                        Top-level location of the source photometry. (default: None)
  --fphotofile FPHOTOFILE
                        Photometric information file. (default: None)
  --emlinesfile EMLINESFILE
                        Emission line parameter file. (default: None)
  --redux_dir REDUX_DIR
                        Optional full path $DESI_SPECTRO_REDUX. (default: None)
  --specproddir SPECPRODDIR
                        Optional directory name for the spectroscopic production. (default: None)
  --uncertainty-floor UNCERTAINTY_FLOOR
                        Minimum fractional uncertainty to add in quadrature to the formal inverse variance spectrum. (default: 0.01)
  --minsnr-balmer-broad MINSNR_BALMER_BROAD
                        Minimum broad Balmer S/N to force broad+narrow-line model. (default: 2.5)
  --debug-plots         Generate a variety of debugging plots (written to $PWD). (default: False)
  --verbose             Be verbose (for debugging purposes). (default: False)

Note

In addition to the Redrock catalog, fastspec requires the DESI coadded spectrum to be located in the same directory (with a default coadd- prefix).

Running fastspec 

To model the spectrum of a single object, provide fastspec the full path of an input Redrock catalog, the targetid of the object of interest, and an output filename:

fastspec $DESI_SPECTRO_REDUX/loa/healpix/sv1/bright/71/7108/redrock-sv1-bright-7108.fits \
  --targetids 39633345008634465 --outfile ./fastspec-example.fits

See the fastspec data model for a full description of the output file. Visualize the results using fastqa:

fastqa ./fastspec-example.fits --outdir ./

_images/fastspec-sv1-bright-7108-39633345008634465.png

The figure above succinctly summarizes the fastspec inputs and modeling results:

Upper-right panel: grz color cutout from the Legacy Surveys centered on the DESI target. The solid and dashed yellow circles represent the \(1.5^{"}\) diameter DESI fiber aperture and a \(10^{"}\) reference aperture, respectively.

Middle-left panel: Three-camera observed DESI spectrophotometry and best-fitting model, shown as light and dark blue, green, and red spectra, respectively, and spanning the observed-frame \(0.36-0.98~\mu m\) wavelength range. The thin, light gray curve around zero flux shows the smooth continuum correction which is added to the thick, dark gray best-fitting stellar population synthesis model (see the algorithms documentation for details).

Top-middle panel: Observed and modeled broadband spectral energy distribution between \(0.1-35~\mu m\) in the observed frame. The orange points (or arrows) show the observed grz (optical) and W1-W4 (infrared) fluxes or \(2\sigma\) upper limits from the Legacy Surveys, and the open square markers represent the photometry synthesized from the best-fitting model. The blue, green, and red spectra in this panel are the best-fitting DESI model after multiplying by the derived aperture correction (shown in the bottom portion of the panel as the factor of 1.32).

Lower-right panel: Zoomed panels showing the data and best-fit model for all the emission lines within the observed spectral range.

Running fastphot 

fastphot models only the broadband photometry at the Redrock redshift, and is significantly faster than fastspec. Using the same example object:

fastphot $DESI_SPECTRO_REDUX/loa/healpix/sv1/bright/71/7108/redrock-sv1-bright-7108.fits \
  --targetids 39633345008634465 --outfile ./fastphot-example.fits

And to generate the QA:

fastqa fastphot-example.fits --outdir ./

_images/fastphot-sv1-bright-7108-39633345008634465.png

Refer to the fastphot data model for a full description of the output file.

Note

The orange points (or arrows) show the observed grz (optical) and W1-W4 (infrared) fluxes or \(2\sigma\) upper limits from the Legacy Surveys, and the open square markers represent the photometry synthesized from the best-fitting model.

Plotting the Best-Fit Model 

The MODELS extension stores the best-fit spectral components, making it straightforward to generate custom figures:

import numpy as np
import fitsio
from astropy.table import Table
import matplotlib.pyplot as plt

from desiutil.dust import dust_transmission
from desispec.io import read_spectra
from desispec.coaddition import coadd_cameras

specfile = '$DESI_SPECTRO_REDUX/iron/healpix/sv1/bright/71/7108/coadd-sv1-bright-7108.fits'
fastfile = 'fastspec-example.fits'

meta = Table(fitsio.read(fastfile, 'METADATA'))
fast = Table(fitsio.read(fastfile, 'FASTSPEC'))

models, hdr = fitsio.read(fastfile, 'MODELS', header=True)
modelwave = hdr['CRVAL1'] + np.arange(hdr['NAXIS1']) * hdr['CDELT1']

spec = read_spectra(specfile).select(targets=meta['TARGETID'])
coadd_spec = coadd_cameras(spec)
bands = coadd_spec.bands[0]

mw_transmission_spec = dust_transmission(coadd_spec.wave[bands], meta['EBV'])

fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
ax1.plot(coadd_spec.wave[bands], coadd_spec.flux[bands].flatten() / mw_transmission_spec,
         color='gray', alpha=0.7, label='Data')
ax1.plot(modelwave, models[0, 0, :], label='Stellar Continuum Model', ls='-', color='blue')
ax1.plot(modelwave, models[0, 1, :], label='Smooth Continuum Correction', ls='--', color='k')
ax1.set_ylim(-2.5, 7.5)
ax1.legend(fontsize=8, loc='upper right')

ax2.plot(coadd_spec.wave[bands], coadd_spec.flux[bands].flatten() / mw_transmission_spec,
         color='gray', alpha=0.7, label='Data')
ax2.plot(modelwave, np.sum(models, axis=1).flatten(), label='Final Model', ls='-', color='red')
ax2.legend(fontsize=8, loc='upper left')
ax2.set_xlabel(r'Observed-frame Wavelength ($\AA$)')

fig.subplots_adjust(hspace=0.05, top=0.95, right=0.95)
fig.text(0.05, 0.5, r'Flux Density ($10^{-17}~{\rm erg}~{\rm s}^{-1}~{\rm cm}^{-2}~\AA^{-1}$)',
          ha='center', va='center', rotation='vertical')

fig.savefig('fastspec-example.png')

Note

All quantities and models are measured from Galactic-extinction-corrected spectra, so the observed data must be dereddened before plotting, as shown above.

Generating and Fitting Stacked Spectra 

Note

Documentation and examples for generating and fitting stacked spectra with stackfit will be added here.

Fitting at Scale 

Target Selection 

The --targetids, --ntargets, --firsttarget, and --mp flags control which objects are fit and how many cores are used. --targetids accepts a comma-separated list:

fastspec $DESI_SPECTRO_REDUX/iron/healpix/sv1/bright/71/7108/redrock-sv1-bright-7108.fits \
  --targetids 39633345008634465,39633334917139798,39633348330522913 \
  --outfile fastspec-example2.fits

To fit a contiguous subset, use --ntargets and optionally --firsttarget:

fastspec $DESI_SPECTRO_REDUX/iron/healpix/sv1/bright/71/7108/redrock-sv1-bright-7108.fits \
  --firsttarget 50 --ntargets 20 --outfile fastspec-example3.fits

When fitting more than one object, enable multiprocessing with --mp:

fastspec $DESI_SPECTRO_REDUX/iron/healpix/sv1/bright/71/7108/redrock-sv1-bright-7108.fits \
  --firsttarget 50 --ntargets 20 --mp 20 --outfile fastspec-example4.fits

Production Runs with mpi-fastspecfit 

For large-scale runs across many healpixels or tiles, use the higher-level mpi-fastspecfit script, which understands the DESI data organization and can distribute work across MPI ranks. One must always specify the spectroscopic production with --specprod.

Use --plan or --dry-run to inspect the work before committing:

mpi-fastspecfit --specprod iron --survey sv1 --program bright \
  --healpix 7108 --outdir-data . --plan

INFO:mpi.py:223:_findfiles: Building file list for survey=sv1 and program=bright
INFO:mpi.py:309:plan: Found 1/1 redrockfiles (left) to do.

mpi-fastspecfit --specprod iron --survey sv1 --program bright \
  --healpix 7108 --outdir-data . --dry-run

INFO:mpi.py:223:_findfiles: Building file list for survey=sv1 and program=bright
INFO:mpi.py:309:plan: Found 1/1 redrockfiles (left) to do.
INFO:mpi-fastspecfit:46:run_fastspecfit: Planning took 0.01 sec
INFO:mpi-fastspecfit:96:run_fastspecfit: Rank 0, ntargets=264: fastspec /global/cfs/cdirs/desi/spectro/redux/iron/healpix/sv1/bright/71/7108/redrock-sv1-bright-7108.fits -o ./iron/healpix/sv1/bright/71/7108/fastspec-sv1-bright-7108.fits --mp 128

To run on a single interactive Perlmutter node:

salloc -N 1 -C cpu -A desi -t 00:10:00 --qos interactive -L cfs
source /global/cfs/cdirs/desi/software/desi_environment.sh main
module load fastspecfit/main
time mpi-fastspecfit --specprod iron --survey sv1 --program bright \
  --healpix 7108 --mp 128 --outdir-data .
ls -l ./iron/healpix/sv1/bright/71/7108

INFO:mpi.py:223:_findfiles: Building file list for survey=sv1 and program=bright
INFO:mpi.py:309:plan: Found 1/1 redrockfiles (left) to do.
INFO:mpi-fastspecfit:46:run_fastspecfit: Planning took 0.16 sec
INFO:mpi-fastspecfit:96:run_fastspecfit: Rank 0, ntargets=264: fastspec /global/cfs/cdirs/desi/spectro/redux/iron/healpix/sv1/bright/71/7108/redrock-sv1-bright-7108.fits -o ./iron/healpix/sv1/bright/71/7108/fastspec-sv1-bright-7108.fits --mp 128
INFO:mpi-fastspecfit:119:run_fastspecfit:   rank 0 done in 113.58 sec
INFO:mpi-fastspecfit:140:run_fastspecfit: All done at Sun Aug  7 06:17:02 2022

real  1m55.770s
user  14m38.856s
sys   1m16.424s

total 12092
-rw-rw-r-- 1 ioannis ioannis 12007670 Aug  7 06:17 fastspec-sv1-bright-7108.fits
-rw-rw-r-- 1 ioannis ioannis   370424 Aug  7 06:17 fastspec-sv1-bright-7108.log

Omitting --survey, --program, or --healpix causes the code to iterate over all available values. For example, to plan a full SV3 run:

mpi-fastspecfit --specprod iron --survey sv3 --outdir-data . --plan
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=bright
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=dark
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=other
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=backup
INFO:mpi.py:309:plan: Found 1023/1023 redrockfiles (left) to do.
INFO:mpi.py:326:plan: Skipping 70 files with no targets.

To fit broadband photometry instead of spectroscopy, add --fastphot:

mpi-fastspecfit --specprod iron --survey sv3 --outdir-data . --plan --fastphot
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=bright
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=dark
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=other
INFO:mpi.py:223:_findfiles: Building file list for survey=sv3 and program=backup
INFO:mpi.py:309:plan: Found 1023/1023 redrockfiles (left) to do.
INFO:mpi.py:326:plan: Skipping 70 files with no targets.

mpi-fastspecfit also supports tile-based coadds via --coadd-type:

mpi-fastspecfit --specprod iron --coadd-type cumulative --tile 80613 --outdir-data . --plan
INFO:mpi.py:309:plan: Found 10/10 redrockfiles (left) to do.

mpi-fastspecfit --specprod iron --coadd-type pernight --tile 80613 --outdir-data . --plan
INFO:mpi.py:309:plan: Found 57/57 redrockfiles (left) to do.

mpi-fastspecfit --specprod iron --coadd-type perexp --tile 80613 --outdir-data . --plan
INFO:mpi.py:309:plan: Found 283/283 redrockfiles (left) to do.

Slurm Batch Jobs 

Note

A Slurm batch job template for multi-node production runs will be added here.

Merging Output Catalogs 

Note

Documentation on merging per-healpix output files into a single catalog will be added here.