LargeSynopticSurveyTelescop
    DataManagementScience
    Design
    J.D.Swinbank,T.Axelrod,A.C.Becker,
    J.F.Bosch,H.Chiang,D.R.Ciardi,A.J.
    G.P.Dubois-Felsmann,F.Economou,M.Fisher-Levine,
    Ivezić,M.Jurić,T.Jenness,R.L.Jones,
    K-T.Lim,R.H.Lupton,F.Mueller,D.Petravick,
    D.J.Reiss,D.Shaw,C.Slater,M.Wood-Vasey
    fortheLSSTDataManagement
    LDM-151
    LatestRevision:2017-07-19
    ThisLSSTdocumenthasbeenapprovedasaContent-Controlled
    ChangeControlBoard.Ifthisdocumentischangedorsuperseded,
    taintheHandledesignationshownabove.Thecontrolisonthe
    withthisHandleintheLSSTdigitalarchiveandnotprintedversions.
    maybefoundinthecorrespondingDMRFC.
    LARGESYNOPTICSURVEYTELESCOPE

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    Abstract
    TheLSSTScienceRequirementsDocument(theLSSTSRD)specifiesasetofdata
    productguidelines,designedtosupportsciencegoals
    bytheLSSTobservingprogram.Followingtheseguidlines,
    dataproductshavebeendescribedintheLSSTDataProducts
    (DPDD),andcapturedinaformalflow-downfromtheSRDviatheLSSTSystemRe-
    quirements(LSR),ObservatorySystemSpecifications(OSS),totheDataManagement
    SystemRequirements(DMSR).TheLSSTDataManagementsubsystem’s
    bilitiesincludethedesign,implementation,deployment
    pipelinesnecessarytogeneratethesedataproducts.
    designofthescientificaspectsofthosepipelines.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    ii

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    ChangeRecord
    VersionDateDescriptionOwnername
    12009-03-26InitialversionasDocument-7396TimAxelrodetal.
    1.22009-03-27MinoreditsTimAxelrod
    1.32009-04-17GeneraleditsandupdatesTimAxelrod
    1.42009-05-08ExplicitreferencetomultifitaddedtoSection
    6.1
    TimAxelrod
    1.52010-02-11Generaleditsandupdates;generatedfrom
    SysMLmodel
    JeffKantor
    22011-08-04ElevatedtoLDMhandle;generalupdatesand
    edits
    TimAxelrod
    32013-10-07UpdatesforconsistencywithFDRbaselineMarioJuric
    2017-05-08MajorreorganizationforDMreplanMarioJuric
    4.02017-05-19ApprovedinRFC-338andreleased.MarioJuric(approval),
    TimJenness(release)
    4.12017-07-19Removedevelopmentcommentary.Thiscon-
    tentwasincludederroneouslyandwasnot
    partoftheapprovedbaseline.
    TimJenness
    Documentcurator:J.D.Swinbank
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    iii

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    Contents
    1Preface1
    2Introduction2
    2.1LSSTDataManagementSystem.....................
    2.2DataProducts..........................
    2.3DataUnits...........................
    2.4SciencePipelinesOrganization.....................
    3AlertProduction7
    3.1SingleFrameProcessingPipeline(WBS02C.03.01)................
    3.1.1InputData.........................
    3.1.2OutputData.........................
    3.1.3InstrumentalSignatureRemoval...................
    3.1.4PSFandbackgrounddetermination..................
    3.1.5Sourcemeasurement......................
    3.1.6PhotometricandAstrometriccalibration................
    3.2AlertGenerationPipeline(WBS02C.03.04)..................
    3.2.1InputData.........................
    3.2.2OutputData.........................
    3.2.3TemplateGeneration......................
    3.2.4Imagedifferencing.......................
    3.2.5SourceAssociation.......................
    3.3AlertDistributionPipeline(WBS02C.03.03)..................
    3.3.1InputData.........................
    3.3.2OutputData.........................
    3.3.3Alertpostagestampgeneration...................
    3.3.4Alertqueuingandpersistance...................
    3.4PrecoveryandForcedPhotometryPipeline..................
    3.4.1InputData.........................
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    iv

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    3.4.2OutputData.........................
    3.4.3ForcedPhotometryonall
    DIAObjects
    .................
    3.4.4 DIAObject Forced
    ...................
    Photometry:
    3.5 Moving Object Pipeline
    ...................
    (WBS 02C.03.06)
    3.5.1 Input
    .........................
    Data
    3.5.2 Output
    .........................
    Data
    3.5.3 Tracklet .
    identification
    .....................
    3.5.4 Precovery and merging
    ..................
    of tracklets
    3.5.5 Linking tracklets
    ...................
    and orbit fitting
    3.5.6 Global.precovery
    ......................
    3.5.7 Prototype Implementation
    ....................
    4 Calibration Products Production
    34
    4.1 Key Requirements
    .........................
    4.2 Inputs
    .............................
    4.2.1 Bias.........................
    Frames
    4.2.2 Gain.........................
    Values
    4.2.3 Linearity
    ..........................
    4.2.4 Darks
    ...........................
    4.2.5 Crosstalk
    ..........................
    4.2.6 Defect
    .........................
    Map
    4.2.7 Saturation
    .......................
    levels
    4.2.8 Broadband
    ........................
    Flats
    4.2.9 Monochromatic
    ......................
    Flats
    4.2.10 CBP
    ..........................
    Data
    4.2.11 Filter Transmission
    .......................
    4.2.12 Atmospheric Characterization
    ...................
    4.3 Outputs from the Calibration Product Pipelines
    39
    ==
    4.3.1 Master
    .........................
    Bias
    4.3.2 Master
    ........................
    Darks
    4.3.3 Master
    ........................
    Linearity
    The contents of this document are subject to configuration control by the LSST
    v

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4.3.4 Master Fringe
    ......................
    Frames
    4.3.5 Master Gain
    .......................
    Values
    4.3.6 Master
    ........................
    Defects
    4.3.7 Saturation
    .......................
    Levels
    4.3.8 Crosstalk
    ..........................
    4.3.9 Master Impure Broadband
    ...................
    Flats
    4.3.10 Master Impure Monochromatic
    .................
    Flats
    4.3.11 Master Pure Monochromatic
    ..................
    Flats
    4.3.12 Master.......................
    PhotoFlats
    4.3.13 Master Low-resolution
    ................
    narrow-band flats
    4.3.14 Pixel
    .........................
    Sizes
    4.3.15 Brighter-Fatter
    ....................
    Coefficients
    4.3.16 CTE Measurement
    .......................
    4.3.17 Filter Transmission
    .......................
    4.3.18 Ghost
    ........................
    catalog
    4.3.19 Spectral
    .......................
    Standards
    4.3.20 Spectrophotometric
    ...................
    Standards
    4.3.21 Astrometric
    ......................
    Standards
    4.4 CBP Control
    ...........................
    4.5 Calibration Telescope.................
    Input Calibration Data
    4.6 Calibration Telescope
    ....................
    Output Data
    4.6.1 Atmospheric
    .....................
    Absorption
    4.6.2 Night Sky
    .......................
    Spectrum
    4.7 Photometric calibration
    ...................
    walk-through
    4.8 Prototype Implementation
    .......................
    5 Data Release Production
    54
    5.1 Image Characterization
    ...................
    and Calibration
    5.1.1 BootstrapImChar
    .......................
    5.1.2 StandardJointCal
    .......................
    5.1.3 RefineImChar
    ........................
    The contents of this document are subject to configuration control by the LSST
    vi

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    5.1.4 FinalImChar
    .........................
    5.1.5 FinalJointCal
    .........................
    5.2 Image Coaddition and..................
    Image Differencing
    5.2.1 WarpAndPsfMatch
    .......................
    5.2.2 BackgroundMatchAndReject
    ....................
    5.2.3 WarpTemplates
    ........................
    5.2.4 CoaddTemplates
    .......................
    5.2.5 DiffIm
    ...........................
    5.2.6 UpdateMasks
    ........................
    5.2.7 WarpRemaining
    ........................
    5.2.8 CoaddRemaining
    .......................
    5.3 Coadd Processing
    .........................
    5.3.1 DeepDetect
    .........................
    5.3.2 DeepAssociate
    ........................
    5.3.3 DeepDeblend
    ........................
    5.3.4 MeasureCoadds
    ........................
    5.4 Overlap Resolution
    .........................
    5.4.1 ResolvePatchOverlaps
    ......................
    5.4.2 ResolveTractOverlaps
    ......................
    5.5 Multi-Epoch Object
    ....................
    Characterization
    5.5.1 MultiFit
    ..........................
    5.5.2 ForcedPhotometry
    .......................
    5.6 Postprocessing
    ..........................
    5.6.1 MovingObjectPipeline
    ......................
    5.6.2 ApplyCalibrations
    .......................
    5.6.3 MakeSelectionMaps
    ......................
    5.6.4 Classification
    .........................
    5.6.5 GatherContributed
    .......................
    The contents of this document are subject to configuration control by the LSST
    vii

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    6 Algorithmic Components
    96
    6.1 Reference Catalog
    .....................
    Construction
    6.1.1 Alert Production
    ..................
    Reference Catalogs
    6.1.2 Data Release Production
    ...............
    Reference Catalogs
    6.2 Instrument Signature
    .....................
    Removal
    6.2.1 ISR for Alert
    .....................
    Production
    6.3 Artifact.........................
    Detection
    6.3.1 Cosmic Ray Identification
    .....................
    6.3.2 Optical
    ........................
    ghosts
    6.3.3 Linear feature detection
    .................
    and removal
    6.3.4 SnapSubtraction
    .......................
    6.3.5 Warped Image....................
    Comparison
    6.4 Artifact Interpolation
    ........................
    6.5 SourceDetection
    .........................
    6.6 Deblending
    ...........................
    6.6.1 Single Frame
    .....................
    Deblending
    6.6.2 Multi-Coadd
    .....................
    Deblending
    6.7 Measurement
    ..........................
    6.7.1 Drivers
    ..........................
    6.7.2 Algorithms
    .........................
    6.7.3 Blended Measurement
    ......................
    6.8 Spatial
    ..........................
    Models
    6.9 Background........................
    Estimation
    6.9.1 Single-Visit Background
    ..................
    Estimation
    6.9.2 Coadd Background
    ...................
    Estimation
    6.9.3 Matched Background
    ...................
    Estimation
    6.10 Build Background
    ......................
    Reference
    6.10.1 Patch
    .........................
    Level
    6.10.2 Tract
    .........................
    Level
    The contents of this document are subject to configuration control by the LSST
    viii

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    6.11 PSF Estimation
    ..........................
    6.11.1 Single CCD.....................
    PSF Estimation
    6.11.2 Wavefront Sensor
    ..................
    PSF Estimation
    6.11.3 Full Visit.....................
    PSF Estimation
    6.12 Aperture.........................
    Correction
    6.13 Astrometric
    .........................
    Fitting
    6.13.1 Single
    .........................
    CCD
    6.13.2 Single
    .........................
    Visit
    6.13.3 Joint........................
    Multi-Visit
    6.14 Photometric
    .........................
    Fitting
    6.14.1 Single CCD
    .......................
    (for AP)
    6.14.2 Single
    .........................
    Visit
    6.14.3 Joint........................
    Multi-Visit
    6.14.4 Large-Scale
    .......................
    Fitting
    6.15 Retrieve Diffim Template
    ....................
    for a Visit
    6.16 PSF Matching
    ...........................
    6.16.1 Image Subtraction
    .......................
    6.16.2 PSF Homogenization
    ..................
    for Coaddition
    6.17 Image Coaddition
    .........................
    6.18 DCR-Corrected Template
    ....................
    Generation
    6.18.1 Generating a DCR Corrected
    .................
    Template
    6.19 Image Decorrelation
    ........................
    6.19.1 Difference Image
    ...................
    Decorrelation
    6.19.2 Coadd Decorrelation
    ......................
    6.20 Star/Galaxy
    .......................
    Classification
    6.20.1 Single.......................
    Frame S/G
    6.20.2 Multi-Source
    .......................
    S/G
    6.20.3
    Object
    Classification
    ......................
    6.21 Variability.......................
    Characterization
    6.21.1 Characterization.................
    of Periodic Variability
    The contents of this document are subject to configuration control by the LSST
    ix

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    6.21.2 Characterization.................
    of Aperiodic Variability
    6.22 Proper Motion and
    DIASources
    Parallax
    .................
    from
    6.23 Association.......................
    and Matching
    6.23.1 Single CCD to Reference
    ...............
    Catalog, Semi-Blind
    6.23.2 Single Visit to Reference
    ...............
    Catalog, Semi-Blind
    6.23.3 Multiple Visits..................
    to Reference Catalog
    6.23.4
    DIAObject
    Generation
    ......................
    6.23.5
    Object
    Generation
    .......................
    6.23.6 Blended Overlap
    ....................
    Resolution
    6.24 Raw Measurement
    .....................
    Calibration
    6.25 Ephemeris .
    Calculation
    .......................
    6.26 Make Tracklets
    ..........................
    6.27 Attribution.......................
    and Precovery
    6.28 Orbit
    ...........................
    Fitting
    6.29 Orbit Merging
    ..........................
    7 Software Primitives
    152
    7.1 Cartesian
    .........................
    Geometry
    7.1.1 Points
    ...........................
    7.1.2 Arrays
    ........................
    of Points
    7.1.3 Boxes
    ...........................
    7.1.4 Polygons
    ..........................
    7.1.5 Ellipses
    ..........................
    7.2 Spherical
    .........................
    Geometry
    7.2.1 Points
    ...........................
    7.2.2 Arrays
    ........................
    of Points
    7.2.3 Boxes
    ...........................
    7.2.4 Polygons
    ..........................
    7.2.5 Ellipses
    ..........................
    7.3 Images
    ............................
    7.3.1 Simple
    ........................
    Images
    The contents of this document are subject to configuration control by the LSST
    x

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    7.3.2 Masks
    ...........................
    7.3.3 MaskedImages
    ........................
    7.3.4 Exposure
    ..........................
    7.4 Multi-Type Associative
    ....................
    Containers
    7.5 Tables
    .............................
    7.5.1 Source
    ..........................
    7.5.2 Object
    ...........................
    7.5.3 Exposure
    ..........................
    7.5.4 AmpInfo
    ..........................
    7.5.5 Reference
    .........................
    7.5.6 .
    Joins
    ..........................
    7.5.7 Queries
    ..........................
    7.5.8 N-Way.
    Matching
    .......................
    7.6 Footprints
    ............................
    7.6.1 PixelRegions
    .........................
    7.6.2 Functors
    ..........................
    7.6.3 Peaks
    ...........................
    7.6.4 FootprintSets
    ........................
    7.6.5 HeavyFootprints
    ........................
    7.6.6 Thresholding
    .........................
    7.7 Basic Statistics
    ..........................
    7.8 Chromaticity
    ........................
    Utilities
    7.8.1 Filters
    ...........................
    7.8.2 .
    SEDs
    ..........................
    7.8.3 Color
    .........................
    Terms
    7.9 PhotoCalib
    ...........................
    7.10 Convolution
    .........................
    Kernels
    7.11 Coordinate Transformations
    ......................
    7.12 Numerical........................
    Integration
    7.13 Random Number.Generation
    .....................
    The contents of this document are subject to configuration control by the LSST
    xi

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    7.14 Interpolation and Approximation
    .................
    of 2-D Fields
    7.15 Common Functions and
    ...................
    Source Profiles
    7.16 Camera Descriptions
    ........................
    7.17 Numerical Optimization
    .......................
    7.18 Monte Carlo
    ........................
    Sampling
    7.19 Point-Spread
    ........................
    Functions
    7.20 Warping
    ............................
    7.21 Fourier Transforms
    .........................
    7.22 Tree Structures
    ..........................
    8 Glossary
    169
    The contents of this document are subject to configuration control by the LSST
    xii

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Data Management Science Pipelines
    1 Preface
    The purpose of this document is to describe the design of
    tions Layer of the Large Synoptic Survey Telescope (LSST)
    include most of the core astronomical data processing software
    The intended audience of this document are LSST software
    presents the baseline architecture and algorithmic selections
    oped to a degree necessary to enable planning and costing
    software development framework. The document assumes the
    quired knowledge of astronomical image processing algorithms
    the state of the art of the field, understanding of the LSST
    has read the LSST
    Science
    SRD
    )Requirements
    as well as the(LSST Data Products
    Document
    DPDD
    ).
    (
    Though under strict changelivingdocument.Firstly,asaconsequence
    control, this is a
    “rollingwave”LSSTsoftwaredevelopmentmodel,thedesigns
    berefinedandmademoredetailedasparticularpipeline
    mented.Secondly,theLSSTwillundergoaperiodofconstruction
    nolessthansevenyears,followedbyadecadeofsurveyoperations.
    uedscientificadequacy,theoveralldesignsandplansfor
    beperiodicallyreviewedandupdated.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    1

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    2Introduction
    2.1LSSTDataManagementSystem
    TocarryoutthismissiontheDataManagementSystem(DMS)
    functions:
    •Processestheincomingstreamofimagesgeneratedby
    servingtoproducetransientalertsandtoarchivethe
    •Roughlyonceperyear,createsandarchivesaDataRelease
    consistentcollectionofdataproductsgeneratedfrom
    dateofsurveyinitiationtothecutoffdatefortheData
    scribedindetailintheDPDD),includemeasurementsoftheproperties
    tions,fluxes,motions,etc.)ofalldetectedobjects,
    sensitivitylimit,astrometricandphotometriccalibration
    log,andlimitedclassificationofobjectsbasedonboth
    producedaswell.
    •Periodicallycreatesnewcalibrationdataproducts,
    thatwillbeusedbytheotherprocessingfunctions,as
    ofthedataproductsabove.
    •MakesallLSSTdataavailablethroughinterfacesthat
    extent,community-basedstandardssuchasthosebeing
    servatory(“VO”),andfacilitatesuserdataanalysis
    dataproductsatDataAccessCenters(“DAC”)andatexternal
    ThisdocumentdiscussestheroleoftheSciencePipelines
    tionslistedabove.Thefourthisdiscussedseparatelyin
    (SUID).
    TheoverallarchitectureoftheDMSisdiscussedinmore
    temDesign(DMSD)document.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    2

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    2.2DataProducts
    TheLSSTdataproductsareorganizedintothreegroups,based
    origin.ThefulldescriptionisprovidedintheDataProductsDPDD);we
    summarizethekeypropertiesheretoprovidethenecessary
    follow.
    •Level1productsareintendedtosupporttimelydetection
    events(variableandtransientsources).Theyaregenerated
    ingthestreamofdatafromthecamerasystemduringnormal
    uctsarethereforecontinuouslygeneratedand/orupdated
    processisofnecessityhighlyautomated,andmustproceed
    humaninteraction.Inadditiontosciencedataproducts,
    “SDQA”1dataproductsaregeneratedtoassessqualityand
    ObservatoryControlSystem(OCS).
    •Level2productsaregeneratedaspartofaDataRelease,
    withanadditionaldatareleaseforthefirst6months
    dataproductsforwhichextensivecomputationisrequired,
    informationfrommanyexposures.Althoughthesteps
    willbeautomated,significanthumaninteractionmaybe
    thequalityofthedata.
    •Level3productsaregeneratedonanycomputingresources
    inanLSSTDataAccessCenter.Often,butnotnecessarily,
    usersofLSSTusingLSSTsoftwareand/orhardware.LSST
    creationofLevel3dataproductsbyprovidingsuitable
    computinginfrastructure,butwillnotbyitselfcreate
    created,Level3dataproductsmaybeassociatedwithLevel
    throughdatabasefederation.Whereappropriate,the
    oftheLevel3creators,mayincorporateuser-contributed
    intotheDMSproductionflow,therebypromotingthemto
    Level1andLevel2dataproductsthathavepassedquality
    bletothedatarightsholdersonacadencedeterminedbythe
    1ScienceDataQualityAnalysis
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    3

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    thesourcecodeusedtogeneratetheseproductswillbemade
    ducibilityandinsightintotheimplementationofalgorithms
    documentation,andalistofreferenceplatformsonwhich
    andexecute.
    Thepipelinesusedtoproducethesepublicdataproductswill
    dataproductsthatmaynotbemadepubliclyavailable(generally
    sededinqualitybyapublicdataproduct).Intermediate
    however,andtheirspecificationisanimportantpartofdescribing
    2.3DataUnits
    Inordertodescribethecomponentsofourprocessingpipelines,
    nomenclaturefortheunitsofdatathepipelinewillprocess.
    Thesmallestdataunitsarethosecorrespondingtoindividual
    ingwithLSSTconventions,weuse“object”torefertothe
    typicallyimpliesaggregationofsomesortoverallexposures),
    alizationofanobjectonaparticularexposure.Inthecase
    justourbestattemptstodefinedistinctastrophysicalobjects,
    definetermsthatrepresentthisprocess.Weuse“family”
    (or,morerarely,sources),and“child”torefertoaparticular
    A“parent”isalsocreatedforeachfamily,representing
    isactuallyasingleobject.Blendsmaybehierarchical;
    thelevelbelow.
    LSSTobservationsaretakenasapairof15-second“snaps”;
    Becausesnapsaretypicallycombinedearlyintheprocessing
    andsurveymodesmaytakeonlyasinglesnap),visitismuch
    forprocessinganddataproducts.Theimagedatafortoavisit
    images.CCD-leveldatafromthecameraisfurtherdatadivided
    aCCD,butthesearealsocombinedatanearlystage,and뜀3CCD“rafts”thatplay
    importantroleinthehardwaredesignarerelativelyunimportant
    visitandCCDthemainidentifiersofmostexposure-level
    Ourconventionfordefiningregionsontheskyisdeliberately
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    4

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    basecapableofworkingwithvirtuallyanypixelization
    schemesmayhavedifferentperformanceorstorageimplications).
    tworegionconcepts:“tracts”and“patches”.Atractis
    coordinatesystem;weassumeitislargerthantheLSSTfield
    essentiallysetbythepointatwhichdistortionintheprojection
    toaffecttheprocessing(bye.g.breakingtheassumption
    thepixelgrid).Tractsaredividedintopatches,allof
    tem.Mostimageprocessingisperfomedatthepatchlevel,
    largelytoensurethatpatch-leveldataproductsandprocessing
    andpatchesaredefinedsuchthateachregionoverlapswith
    lapregionsmustbelargeenoughthatanyindividualastronomical
    inatleastonetractandpatch.Inapatchoverlapregion,
    mericallyequivalent(i.e.equaluptofloatingpointround-off
    overlaps,thisisimpossible,butweexpecttheresultsto
    largertractsandpatchesthusreducestheoverallfraction
    regionsandmustbeprocessedmultipletimes,whileincreasing
    processingindividualtractsandpatches.
    2.4SciencePipelinesOrganization
    LSSTdataprocessingneedsmaybebrokendownintothree
    tion,CalibrationProductsProduction,andDataRelease3,4,and5,
    respectively,wedescribebreakingthesedownintoconstituentpipelines.Inthisdocument,
    apipelineisahigh-levelcombinationofalgorithmsthat
    productioninwhichitisrun.Forinstance,whileboth
    Productionwillincludeapipelineforsingle-visitprocessing,distinct,
    becausethedetailsoftheirdesigndependverymuchonthe
    PipelinesarelargelycomposedofAlgorithmicComponents:
    weexpecttoreuse(possiblywithdifferentconfiguration)
    componentsconstitutethebulkofthenewcodeandalgorithms
    ProductionandDataReleaseProduction,andarediscussed6.Mostalgorithmic
    componentsareapplicabletoanysortofastronomicalimaging
    tomizedforLSST.
    Thelowestlevelinthisbreakdownismadeupofourshared
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    5

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    thatprovideimportantdatastructuresandlow-levelalgorithms,
    ordinatetransformations,andnonlinearoptimizers.Much
    astronomy-related,butessentiallynoneofitisspecific
    makeuseofthird-partylibrarieswheneverpossible.These
    toaccessandprocessLevel1andLevel2dataproductswithin
    LSSTSciencePlatformandassociatedcomputingservices,
    maticrepresentationofthosedataproducts.Sharedsoftware
    section7.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    6

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    3AlertProduction
    AlertProductionisruneachnighttoproducecatalogsand
    ormovedrelativetoapreviousobservation.Thedataproducts
    aregiveninTable1.
    NameAvailabilityDescription
    DIASourceStoredMeasurementsfromdifferenceimageanalysis
    posures.
    DIAObjectStoredAggregatequantitiescomputedbyassociating
    catedDIASources.
    DIAForcedSourceStoredFluxmeasurementsoneachdifferenceimage
    aDIAObject.
    SSObjectStoredSolarsystemobjectsderivedbyassociatingDIASourcesandinfer-
    ringtheirorbits.
    CalExpStoredCalibratedexposureimagesforeachCCD/visit
    snaps)andassociatedmetadata(e.g.WCSand
    ground).
    TemplateCoaddTemporaryDCRcorrectedtemplatecoadd.
    DiffExpStoredDifferencebetweenCalExpandPSF-matchedtemplate
    VOEventStoredDatabaseofVOEventsasstreamedfromtheAlert
    TrackletsPersistedIntermediatedataproductforthegenerationSSObjectsgen-
    eratedbylinkingmovingsourceswithinagiven
    Table1:Tableofderivedandpersisteddataproductsproduced
    detaileddescriptionofthesedataproductscanbefoundin
    Document[LSE-163].
    AlertProductionisdesignedasfiveseparatecomponents:
    eration,alertdistribution,precoveryphotometry,and
    ofthesecomponentsrunasalinearpassthroughofthedata.
    isrunindependentlyoftherestofthealertproduction.
    systemisshowninFigure1.
    Inthisdocumentwedonotaddressestimationoftheselection
    throughtheinjectionofsimulatedsources.Suchaprocess
    modeaspartoftheDRP.Sourcedetectionthresholdscan
    skysources(PSFphotometrymeasurementspositionedin
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    7

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    Figure1:Thealertproductionflowofdatathroughtheprocessing
    processing,alertgeneration,alertdistribution,precovery
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    8

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    3.1SingleFrameProcessingPipeline(WBS02C.03.01)
    TheSingleFrameProcessing(SFM)Pipeline(seeFigure2)isresponsibleforreducing
    orcamera-correctedimagedatatocalibratedexposures,thedetectionandmeasurement
    Sources(usingthecomponentsfunctionallypartoftheObject
    characterizationofthepoint-spread-function(PSF),and
    lutionforanimage.Calibratedexposuresproducedbythe
    informationnecessaryformeasurementofsourceproperties
    terizationalgorithms.
    Astrometricandphotometriccalibrationrequiresthedetection
    ertiesofSourcesonaCCD.AccuratecentroidsandfluxesfortheseSourcesrequireanestima-
    tionofthePSFandbackground,whichinturnrequiresknowledge
    Sourcesonanimage.TheSFMpipelinewill,therefore,iterate
    (see3.1.4)andsourcemeasurement(see3.1.5)
    TheSFMpipelinewillbeimplementedasaflexibleframework
    canbeaddedwithoutmodifyingthestackcode(thiswould
    crosstalkcorrectedimagesshouldanetworkoutagebetween
    resultinonlytherawdatabeingavailable).Thepipeline,
    becapableofbeingrunatthetelescopefacilityduring
    3.1.1InputData
    RawCameraImages:Amplifierimagesthathavebeencorrectedfor
    thecamerasoftware.Allimagesfromavisitshouldbeavailable
    AnapproximateWCSisassumedtobeavailableasmetadata
    ControlSystemwithanabsolutepointinguncertainty(forOSS-REQ-0298
    absPointErr
    andthefieldrotationknowntoanaccuracyof32arcsecondsLTS-206].
    ReferenceDatabase:Afull-skyastrometricandphotometricreference
    rivedeitherfromanexternaldataset(e.g.Gaia)orfrom
    thecurrentGaiadatareleasetimelinetheinitialreference
    trometricuncertaintyof㴀㄀⼀㘀milliarcsecondsandaphotometricuncertainty㴀20millimag
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    9

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    Figure2:Singleframeprocessingofthenightlydata:instrument
    trometricandphotometriccalibration,backgroundandPSF
    correctedcameraimages.
    (fora봉㸀㈀㨀G2Vstar).Theexpectedreleaseofthesecalibration
    derivedfromtheGaiaspectrophotometricobservationsof
    CalibrationImages:Flat-fieldcalibrationimagesforallpassbands
    ateforthetimeatwhichtheobservationswereundertaken.
    intheflat-fieldsfornon-uniformpixelsizes-theflat-fields
    brightness.AflatSEDwillbeassumedforallflatfieldcorrections.
    imagesscaledtoanamplitudederivedfromtheskybackground
    available).
    ImageMetadata:ListofthepositionsandextentsofCCDdefects
    focalplane;electronicparametersforallCCDs(saturation
    tronicandphysicalfootprintfortheCCDs,linearityfunctions,
    PSFwidthwithsourcebrightness(brighter-fatter),and
    basedWCS(e.g.aseriesofopticaldistortionmodels)as
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    10

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    3.1.2OutputData
    CalExpImages:Acalibratedexposure(CalExp)isanExposureobject.TheCalExpcontains
    theimagepixelvalues,avarianceimage,abitwisemask,
    (possiblydecomposedintoseparablecomponents),aphotometric
    modelforthebackground.Forthealertproduction,itis
    per-pixelcovariancewillbepersistedbutthiswillberevisited
    ofimagesubtractionandanomalycharacterizationasdescribed3.2
    SourceDatabases:AcatalogofSourceswithmeasuredfeaturesdescribed3.1.5
    OCSDatabaseAparameterizationofthePSF,WCS,photometric
    eachCCDinavisit.ThePSFmaybeasimplifiedversion(e.g.
    fortheAlertproduction.Thesedatawillbemadeavailable
    (TCS)toassessthesuccessofeachobservation.Alimited
    onthesummittogeneratethisinformationorthedatawill
    thedatacenterthatwillbeaccessibletotheTCS.
    3.1.3InstrumentalSignatureRemoval
    InstrumentalSignatureRemovalcharacterizes,corrects,
    raw)amplifierimagestogenerateaflat-fieldedandcorrected
    PipelineTasks
    •Masktheimagedefectsattheamplifierlevelbasedonthe
    CCDsaturationlimits
    •Assembletheamplifiersintoasingleframe(maskingmissing
    •Applyfullframecorrections:darkcurrentcorrection,
    ness,fringecorrections.Flatfieldswillassumeaflat
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    11

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    forthesource.Fringeframeswillbenormalizedbyfitting
    ground.
    •Applypixellevelcorrections:applyacorrectionmodel
    thePSF,correctforstaticpixelsizeeffectsbasedon
    •Interpolateacrossdefectsandsaturatedpixelsassuming
    nominalFWHM).AnestimateofthePSFwillbeneededfor
    S/OCS)orinterpolationmaybeneededtobeperformed3.1.4
    •Applyacosmicraydetectionalgorithmasdescribedin6.3.1
    •Generateasummedanddifferenceimagefromtheindividual
    unionofthemaskpixelsineachsnap
    DependentonthepropertiesofthedeliveredLSSTimagequality
    requiredtomodelanybulkmotionbetweensnapspriorto
    orthegroundlayerdominatethelowerordercomponentsof
    3.1.4PSFandbackgrounddetermination
    GivenexposuresthathavebeenprocessedthroughInstrumentSources
    mustbedetectedtodeterminetheastrometricandphotometric
    Asnotedpreviouslyaniterativeprocedurewillbeadopted
    backgroundandPSF,andtocharacterizethepropertiesof
    criteriaforthisprocedurearenotcurrentlydefined.The
    threeiterations.
    PipelineTheTasksiterativeprocessforPSFandbackgroundestimation
    •BackgroundestimationonthescaleofasingleCCDisas6.9,whichdivides
    theCCDintosubregionsandestimatesthebackground
    sourcepixels.
    •Subtractionofthebackgroundandthedetectionofsources6.5The
    initialdetectionthresholdforsourcedetectionwill栌,with栌estimatedfromvariance
    imageplane.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    12

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    •Measurementofthepropertiesofthedetectedsources3.1.5).Dependentonthe
    densityofsourcesitmaybenecessarytodeblendtheimages6.6
    •SelectionofisolatedPSFcandidatestarsbasedonasignal-to-noise
    50栌).Thisthresholdissignificantlydeeperthanthemagnitude
    riccatalogsbutisthethresholdatwhichtheastrometric
    photonnoiseislessthan10masandthephotometricnoise
    oftheuseofadeeperDRPderivedreferencecatalog.
    •SingleCCDPSFdeterminationusingthetechniquesdescribed6.11.1andtheselected
    brightsources
    •MaskingofsourcepixelswithintheCCD(growingthefootprintSourcestomask
    theouterregionsoftheSourceprofileswilllikelyberequiredtoexclude
    thebackgroundfromlowsurfacebrightnessfeatures).
    Thedefaultexpectationisthatalltaskswithinthisprocedure
    TheremaybesignificantspeedoptimizationstobegainedSourcedetection
    stepafteraninitialdetectionifthenumberofsources
    datestothebackgroundmodel.
    3.1.5Sourcemeasurement
    FortheSourcecataloggeneratedin3.1.4,sourcepropertiesaremeasuredusing
    featuresdescribedin6.7SourcemeasurementisforallsourceswithinSourcecatalogand
    notjustthebrightsubsetusedtocalibratethePSF.Weanticipate
    algorithmswithintheSourcemeasurementstep,
    •CentroidsbasedonastaticPSFmodelfit(see6.7.2and6.7.2)
    •Aggregationofpixelflagsasdescribedin6.7.2
    •AperturePhotometryasgevenin6.7.2(butonlyforoneortworadii)
    •PSFphotometrygivenin6.7.2assumingastaticPSFmodelfit
    •AnaperturecorrectionestimatedassumingastaticPSF
    curveofgrowthfordetectedsourcesasgivenin6.12
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    13

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    3.1.6PhotometricandAstrometriccalibration
    Photometricandastrometriccalibrationentailsa“semi-blind”
    ingofthetelescopeisknowntoanaccuracyof2arcseconds)
    eitherfromtheDRPObjectsorfromanexternalcatalog(see3.1.1),thegenerationofa
    (onthescaleofaCCDorfullfocalplane),andthegeneration
    thescaleofaCCD).Thesealgorithmsmustdegradegracefully
    errors(e.g.duringtheinitialcalibrationofthesystem
    operateina“blind”modewherethepointingandorientation
    PipelineTheTasksphotometricandastrometriccalibrationis
    thescaleofasingleCCD.Itispossiblethatthecalibration
    largerscales(uptoafullfocalplane)ifthereissignificant
    point,orifastrometricdistortionscannotbecalibrated
    accuracy(i.e.theastrometricdistortionsdonotdominate
    subtraction).Afullfocalplanelevelcalibrationstrategy
    withintheprocessingoftheCCDsasthedetectionsonall
    priortotheastrometricfit.
    Theproceduresusedtomatchandcalibratethedataare,
    •CCDlevelsourceassociationbetweentheDRPreference
    andSourcesdetectedduringthePSFandbackgroundestimation
    fiedOptimisticBapproachdescribedin6.23.1Givenanastrometricaccuracy㴀㄀⼀㘀
    milliarcsecondsfromexternalcatalogssuchasGaia봉㸀㈀㨀G2Vstar)oranac-
    curacyof㴀㘀㄀milliarcsecondsfortheDRPcatalogsthesearch
    dominatedbytheuncertaintiesinthepointingofthe
    ofthecamera.
    •Generationofaphotometricsolutiononthescaleofa6.14.1
    •FittingofaWCSastrometricmodelforasingleCCDusing6.13.1
    TheWCSmodelisexpectedtobecomposedofasumoftransforms
    ponents(e.g.aopticalmodelforthetelescope,alookup
    suchastreerings).
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    14

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    •Persistanceoftheastrometric,PSF,andphotometric
    TelescopeControlsystem(TCS)(see3.1.2)
    GiventhenumberofstarsavailableonaCCDorthecomplexity
    fortheLSST(e.g.thedecompositionoftheWCSintocomponents)
    theastrometricandphotometricsolutionsbeperformed
    CCD.Forthesecasesthealgorithmsusedwillbesinglevisit6.23.2),singlevisit
    photometricsolutions(see6.14.1),andsinglevisitastrometricfits(see6.13.2).Fittingtoafull
    focalplaneintroducesasynchronizationpointinthealert
    havecompletedtheirpreviousprocessingstepspriorto
    Astrometricandphotometricsolutionswithincrowdedfields
    isolatedsourceswithinaCCDimage.TheorderoftheWCS
    therefore,dependonthenumberofcalibrationSourcesthatareavailable.
    3.2AlertGenerationPipeline(WBS02C.03.04)
    TheAlertGenerationpipelineidentifiesvariable,moving,
    ibratedexposurebysubtractingadeepertemplateimage3).TheDIASourcesde-
    tectedonaDiffExpareassociatedwithknownDIAObjectsandSSObjects(thathavebeenprop-
    agatedtothedateoftheCalExpexposure)andtheirproperties
    imagedifferencingrequiresthecreationorretrievalof
    astrometryandPSFoftheTemplateCoaddtoaCalExp,and
    fromtheCalExp.SpuriousDIASourceswillberemovedusingmorphological
    basedclassificationalgorithms.
    TheAlertGenerationpipelineisrequiredtodifference,DIASource
    sourceswithin24s(allowingformultiplecoresandmultithreading
    3.2.1InputData
    CalExpImages:Calibratedexposureprocessedthrough3.1withassociatedWCS,PSF,
    variance,andbackgroundestimation.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    15

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    Figure3:Generationofalertsfromthenightlydata:imagedifferencing
    ofthepropertiesoftheDIASources,identificationandfilteringofspuriousevents,
    ofpreviouslydetectedDIAObjectsandSSObjectswiththenewlydetectedDIASources.
    CoaddImages:TemplateCoaddimagesthatspatiallyoverlapwith
    cessedthrough3.1Thiscoaddedimageisoptimizedforimagesubtraction
    tobecharacterizedintermsofatract/patch/filter.Generation
    fordifferentialchromaticrefractionorbegeneratedfor
    andparallacticangles.
    ObjectDatabases:ObjectsthatspatiallyoverlapwiththeCalExpimages
    3.1ThisObjectcatalogwillprovidethesourcelistfordetermining
    detectedDIASources.
    DIAObjectDatabases:DIAObjectsthatspatiallyoverlapwiththeCalExp
    through3.1ThisDIAObjectcatalogwillprovidetheassociationlistDIA-
    Sourceswillbematched.
    SSObjectDatabases:TheSSObjectlistatthetimeoftheobservation.SSObjectposi-
    tionswillbepropagatedtothedateoftheCalExpobservations
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    16

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    tionlistforcross-matchingagainstthedetectedDIASourcestoidentifyknownSolarSystem
    objects.
    Referenceclassificationcatalogs:ClassificationofDIASourcesbasedontheirmorpholog-
    icalfeatures(andpossiblyestimatesofthelocaldensity
    DIASource)willbeundertakenpriortoassociationinorderto
    itives.Thedatastructuresthatdefinetheseclassifications
    spuriousnessanalysis.
    3.2.2OutputData
    DiffExpImages:differencesderivedbysubtractingaTemplateCoadd
    image.
    DIASourceDatabases:DIASourcesdetectedandmeasuredfromtheDiffExps
    ofparametersdescribedinDPDDwillbepersisted.
    DIAObjectDatabases:DIASourcewillbeassociatedwithexistingDIAObjectsandpersisted.
    NewDIASource(i.e.thosenotassociated)willgenerateanewinstanceDIAObject.
    3.2.3TemplateGeneration
    Templategenerationrequiresthecreationorretrieval6.15)ofaTemplateCoaddthat
    ismatchedtothepositionandspatialextentoftheinput
    plateCoaddcouldbefromapersistedCoaddthatwasgenerated
    comparable(withinapredefinedtolerance)airmassandparallactic
    thatcorrectsfortheeffectofdifferentialchromaticrefraction6.18).Itisexpectedthat
    theseoperationswouldbeundertakenonaCCDlevelbutfor
    mightbereturnedforafullfocalplaneoraseriesofpatchesortract.
    PipelineTasks
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    17

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    •QueryforaTemplateCoaddimagesthatarewithinagiven
    (default2years)ofthecurrentCCDimage,andarewithin
    lacticangle.
    •(optional)DeriveaseeingandDCRcorrectedTemplateCoadd
    templategenerationin6.18).Thecurrentprototypeapproachassumes
    plateCoaddwillbederivedforthezenithandwillcomprise
    wavelengthdimensions(alowresolutionspectrumper
    vationwillrequirealigningtheDCRcorrectioninthe
    theCalExp.
    3.2.4Imagedifferencing
    ImagedifferencingincorporatesthematchingofaTemplateCoadd
    andintermsofimagequality),subtractionofthetemplate
    mentDIASourcesof,removalofspuriousDIASources,andassociationoftheDIASourceswith
    previouslyidentifiedDIAObjects,andSSObjects.
    PipelineTasks
    •DeterminearelativeastrometricsolutionfromtheWCS
    andCalExpimage
    •MatchtheDRPSourcesfortheTemplateCoadd(see5.1.4)againstSourcesfromtheSFM
    pipeline(see3.1)oftherawimages.
    •WarporresampletheTemplateCoaddusingaLanczosfilter7.20)to
    matchtheastrometryoftheCalExp.Itispossiblethat
    plateCoaddandCalExpusingfaintsourcewillneedto
    accuracyoftheWCS.
    •ForCalExpimageswithanimagequalitythatisbetter
    volvetheCalExpimagewiththePSF.Useaconvolution7.10)thatismatched
    tothesourcedetectionkernel.Thisreducestheneed
    matching(see6.16.1)
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    18

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    •MatchthePSFoftheCalExpandTemplateCoaddimagesas6.19.1andcon-
    structaspatialmodelforthematchingkernel.Thisapproach
    acommonPSFthroughhomogenizationofthePSF(see6.16.2
    •ApplythematchingkerneltotheTempCoaddandsubtract
    DiffExp(asdescribedin6.16.1).Dependentontherelativesignal-to-noise
    andtemplateimagedecorrelationofthetemplateimage
    templatewithamatchingkernelmaybenecessary(see6.19.1)
    •DetectDIASourcesontheDiffExpusingthealgorithmsdescribed6.5Convolutionwith
    adetectionkernelwilldependonwhethertheCalExpwas
    •MeasurementsoftheDIASourcesontheDiffExpwillincludedipolemodels
    PSFmodels(see6.7.2and6.7.2andparametersdescribedinTable2ofDPDD.The
    specificalgorithmsusedformeasurementofDIASourceswilldependonwhether
    CalExpimagewaspreconvolved.
    •MeasurementofthePSFfluxonsnapdifferenceimagesforDIASources.
    •Theapplicationofspuriousnessalgorithms,alsoknown
    atthistimedependentonwhetherthenumberoffalsepositives
    detectedsources(see6.7.2)2.DIASourcesclassifiedasspuriousatthisstagemopsPurityMin
    OSS-REQ-0354
    persisted(dependentonthedensityofthefalsepositives).
    basedonatrainedrandomforestclassifier.Itislikely
    willneedtobeconditionedontheimagequalityandairmass
    3.2.5SourceAssociation
    InSourceAssociationDIASourcesdetectedwithinagivenCCDwillbecross-matched
    ated(see6.23.4)withtheDIAObjecttableandtheSSObjects(whoseephemerideshave
    generatedforthetimeofthecurrentobservation).The
    accountfortheuncertaintieswithinthepositions.The
    orsonexpectedpropermotionsforthesources.External
    eventsfromothertelescopesorinstruments)canbeincorporated
    thenightlypipeline(essentiallytreatingexternalsourcesDIAObjectsandasso-
    ciatingthemwiththeDIASources)enablingeithermatchingtoDIASourcesorgenerationof
    forcedphotometryatthepositionoftheexternalsource.
    2Therequirementfora50%falsepositiverateisgiveninthe
    OSS(whendiscussingSolarSystemObject
    ments)andimpactsthesizingmodelforthealertstream
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    19

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    PipelineTasks
    •GeneratethepositionsofSSObjectsthatoverlapaDiffExpgivenitsobservation
    propagatingtheSSObjectorbit(see6.25)
    •Asdescribedin6.23.4sourceassociationwillbeundertakenforDIASources.Matching
    willbetoDIAObjects,andtheephemeridesofSSObjects.PositionsforDIAObjectswillbe
    basedonaatimewindowed(default30day)averageofDIASourcesthatmakeup
    theDIAObject.Alinearmotionmodelforparallaxandproper
    topropagatetheDIAObjecttothetimeoftheobservation.Aprobabilistic
    mayneedtoaccountforone-to-manyandmany-to-oneassociations.
    itmaybenecessarytogeneratejointassociationsacrossDIAObjects(andassociated
    DIASources)inthelocalvicinityofaDIASourcetocorrectformis-assignment
    ousobservations.ThiscouldincludethepruningandDIASourcesbe-
    tweenDIAObjects.Abaselineapproachfornightlyprocessing
    amaximumaposterioriestimatefortheassociation.
    •DIASourceswillbepositionallymatchedtothenearest3stars
    Objectdatabase.Initssimplestcasethesearchalgorithm
    neighborsearch(thedefaultradiusforassociationisObjects
    willbepersistedasameasureoflocalenvironment.
    •DIASourcesunassociatedwithaDIAObjectwillinstantiateanewDIAObject.
    •TheaggregatepositionsfortheDIAObjectswillbeupdatedbasedonarolling
    dow(default30days).
    •PropermotionandparallaxoftheDIAObjectwillbeupdatedusingalinear
    describedin6.22
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    20

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    3.3AlertDistributionPipeline(WBS02C.03.03)
    TheAlertDistributionPipelinetakesthenewlydiscoveredDIAObjects(includingtheirassoci-
    atedhistoricalobservations)andallrelatedmetadataDPDD,anddelivers
    alertpacketsinVOEventformattoavarietyofendpointsviastandardVO-
    EventTransportProtocol;VTP).Packagingoftheeventwill
    stampcutouts(30x30pixelsonaverage)forthedifference
    togetherwiththevarianceandmaskpixelsforthesecutouts.
    TheSRDrequiresthatthedesignoftheLSSTalertsystemshould
    㠀events
    pernight,whichcorrespondsto10
    㔀alertspervisitor50alertsperCCD(with
    subsequentvisitsaveraging39seconds).Allalerts(up
    㔀pervisit)mustbetransmitted
    within60softheclosureoftheshutterofthefinalsnapwithin
    Foranightlyeventrateof10
    㠀,andassumingtheschemadescribedinTables
    DPDDtogetherwiththegenerationofthepostagestampcutouts,VOEvents
    datastreamamountstoapproximately600GBofdatapernight
    data).TheAlertDistributionpipelineisdesignedtodistribute
    includingtheaccesspointofexternaleventbrokers,shown4.
    InadditiontothefulldatastreamtheAlertGeneration
    teringservice.ThisservicewillrunattheLSSTU.S.Archive
    tronomerstocreatefilters(see3.3.4)thatlimitwhatalerts,andwhatfields
    areultimatelyforwardedtothem.Theseuserdefinedfilterswillbeconfigurablewith
    fiedSQL-likedeclarativelanguage.Accesstothisfiltering
    byauser.
    VOEventalertswillbepersistedinanalertdatabaseaswell
    queue.Thealertdatabase(AlertDB)willbesynchronized
    bequeriablebyexternalusers.Themessagequeuethatdistributes
    havethecapabilitytoreplayeventsforthecaseofabreak
    thequeueandclientbutnottosupportgeneralqueries.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSST
    21

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSciencePipelinesDesignLDM-151LatestRevision2017-07-19
    Figure4:Distributionofalertsfromthenightlyprocessing:
    aroundeachDIASourcedetected,distributionoftheDIAObjectsasVOEvents,simplefiltering
    oftheeventstream,andpersistenceoftheeventsinadatabase.
    3.3.1InputData
    DIAObjectDatabase:DIAObjects,withnewDIASources,generatedthroughimagedifferenc-
    ingwillbeusedtocreatealertpackets.
    DifferenceImages:TheDiffExpwillbeusedtogeneratepostagestamp
    DIASourceswithintheCCD.
    CoaddImages:TheTemplateCoaddusedinimagesubtractionwill
    postagestampimagesofthetemplateimageforDIAObjects.
    3.3.2OutputData
    VOEvent
    Database:
    VOEvents
    generated from
    DIAObjects
    the
    and cutouts will be persisted
    within a database (e.g. a noSQL database) or object store.
    3.3.3 Alert postage stamp generation
    Creates the associated image cutouts (30x30
    DIAObjects
    pixels
    (cutouts
    on average)
    are generated from the current observation and not from historical
    The contents of this document are subject to configuration control by the LSST
    22

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Pipeline Tasks
    •Extract from the DiffExp
    DIAObject
    the cutout
    with
    DIASource
    a of each
    detected within
    the current observation. Cutout images
    DIASource
    will
    but
    be
    on
    scaled
    average will be 30x30 pixels. Variance and mask planes,
    sociated metadata will be persisted. The prototype implementation
    cutouts will be persisted as FITS images with a projection
    the DiffExps.
    •Extract from the TemplateCoadd
    DIAObject
    a
    with
    cutout
    DIASource
    a
    of
    detected
    each
    within the current observation. Cutout images will be
    those derived from the DiffExp. Variance and mask planes,
    data will be extracted with the pixel data. The prototype
    these cutouts will be persisted as FITS imagesand that
    DiffExps.
    3.3.4 Alert queuing and persistance
    The alert queue distributes
    DIAObject
    with
    andnew
    DIASources
    persists
    as
    VOEvents
    through
    a message queue.
    limited
    It includes
    filteringainterface but
    VOEvents
    persists
    in an
    the
    AlertDB. The event message stream and the AlertDB will be
    24 hours.
    Pipeline Tasks
    •Publish
    DIAObjects
    to a caching
    message
    Apache
    queue
    Kafka
    ) through
    (e.g.
    the butler.
    The prototype implementation assumes a distributed
    tem
    thatpublication-subscription
    uses a
    model for communication between
    the queue. This model maintains feeds of messages in categories
    ample topic would
    DIAObject
    .
    beWhether
    a
    a topic would
    DIAObject
    comprise
    or
    a full
    a subset of the data remains open (passing subsets of parameters
    would require that the client be able to synchronize
    DIAObject
    ).
    and
    For each of the 189 CCDs, approximately 50 events will
    messaging queuing system. The distribution of the events
    The contents of this document are subject to configuration control by the LSST
    23

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    be synchronized with other CCDs within the focal plane
    independently processed).
    •A consumer layer will subscribe to the message
    VOEvents
    queue and
    and distribute these events to external users. To allow
    message queue and the consumer the message queue must
    events.
    •The consumer layer will provide a command line API to define
    the events (limited toDIAObject
    querying
    fields,
    on existing
    or filtering the attributes
    the
    DIAObject
    ). Web-based interfaces to the consumer layer
    •Filtered or the
    DIAObjects
    full stream
    will beof
    packaged
    VOEvents
    and
    into
    broadcast to
    VOEvent clients through the consumer layer
    •A full, unfiltered, VOEvent alert stream will be broadcast
    sumer layer.
    •Prior to the start of the subsequent night’s observations,
    flushed and synchronized with the AlertDB. It is possible
    on longer timescale but it is a requirement hat synchronization
    hours of the observations.
    To cope with the variation in density of events as a function
    need for fault tolerance the message queue will need to be
    data. Given the 600GB of data generated per night
    DIA-
    from the
    Object
    stream will require about 0.1 Gb/s network capacity.
    instantiate a new consumer for each filter
    VOEvents
    (or
    from
    client)
    a sin-or will
    gle subscription to the message queue is an open question
    network topology (internal and external to the data center
    The AlertDB will have an interface that can be queried (to
    including searches on other than timestamps. It is expected
    datastore (e.g. Cassandra).
    The contents of this document are subject to configuration control by the LSST
    24

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    3.4 Precovery and Forced Photometry Pipeline
    The precovery and forced photometry pipeline
    5). Forced
    performs
    PSF two
    photometry is undertaken
    DIAObjects
    that
    forhave
    all a detected
    DIASource
    within a, default
    1 year, window of time from the observation. Second, within
    tometry is performed on
    DIASources
    all unassociated
    within an image
    DIAObjects
    (i.e.
    ). For
    new
    each new
    DIAObject
    , forced (PSF) photometry will be measured at the
    in each of the preceding 30-days of DiffExps.
    Forced photometry is not required prior to alert generation.
    photometry is required within 24 hours of the completion
    precovery can be undertaken as part of the nightly workflow
    required to distribute the alerts.
    Figure
    Forced
    5:
    photometry
    DIAObjects
    : forced photometry on a night’s DiffExp
    DIAObjects
    that have detected
    DIASources
    within the last year, precovery photometry
    previous 30 days of DiffExps
    DIAObjects
    for new
    3.4.1 Input Data
    Difference images:
    A cache of DiffExps within a finite time interval
    previous nights observations (inclusive of the previous
    The contents of this document are subject to configuration control by the LSST
    25

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    DIAObject Database:
    All
    DIAObjects
    with
    DIASource
    a
    detection within the last 12
    and all unassociated
    DIAObjects
    observed
    (new)
    within the previous night
    3.4.2 Output Data
    DIAForcedSource
    Forced
    Databases:
    PSF photometry at the centroid (from
    individual
    DIASource
    centroids)
    DIAObject
    of.aThe forced photometry is undertaken
    current night’s
    DIAObjects
    DiffExp
    with
    for
    DIASources
    all
    detected within the last year,
    the previous 30 days of DiffExp
    DIASources
    for all
    .
    newly detected
    3.4.3 Forced Photometry
    DIAObjects
    on all
    Generate forced (PSF) photometry
    DIAObjects
    on
    that
    theoverlap
    DiffExp for
    with
    all
    the
    print of the CCD. Forced photometry
    DIAObjects
    isfor
    only
    which
    generated
    there has
    for
    been
    DIASource
    a
    detection within the last 12 months. The forced
    the forced photometry table in the Level 1 database. Alerts
    tion of forced photometry and forced photometry is not released
    means that this component of the processing is not subject
    quirements for nightly processing.
    Pipeline Tasks
    •Extract
    DIAObjects
    allwithin the Level 1 database
    DIASource
    within
    a detected
    the
    last year (including the current nights observations).
    the
    DIASource
    and
    DIAObject
    association.
    •For the aggregate positions
    DIAObject
    undertake
    within the
    a PSF forced measurement
    as described6.7.1
    in section
    •Update the forced photometry tables in the Level 1 database.
    3.4.4 DIAObject Forced Photometry:
    Updated forced photometry
    DIAObjects
    table for all new
    The contents of this document are subject to configuration control by the LSST
    26

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Pipeline Tasks
    •Extract from the Level
    DIAObjects
    1 database
    that were
    all
    unassociated
    DIA-
    (i.e.
    Source
    detections) from the previous nights
    DIAObjects
    reduction.
    will
    need to account for cases
    DIASources
    where
    are observed
    new
    more than once within
    night (where the second or subsequent observations
    DIAObject
    ).
    do
    •Extract DiffExps within a default 30 day window prior to
    •Force photometer the extracted
    6.7.1
    images
    using aas
    PSF
    described
    model andin
    the
    centroid defined
    DIAObject
    in the
    •Update the forced photometry table within the Level 1
    The contents of this document are subject to configuration control by the LSST
    27

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    3.5 Moving Object Pipeline (WBS 02C.03.06)
    The Moving Object Pipeline (MOPS) is responsible for generating
    tem data products. These are Solar System objects with associated
    and detected
    DIASources
    . Quantitatively, it shall be capable
    orbitCompleteness
    of detecting
    tem objects that meet the
    criteria
    OSS
    (i.e. the
    specified
    observations
    in the OSS-REQ-0159
    required
    an orbit). Each visit within 10 degrees of the Ecliptic will
    Components of MOPS are run during and separately
    6).
    from nightly
    MOPS for nightly processing
    3.2.5
    asis
    part
    described
    of source
    inassociation.
    processes newly
    DIAObjects
    detected
    to search for candidate asteroid tracks.
    for Day-MOPS
    DIASource
    is to detections
    link
    within a night (called tracklets),
    tracklets across multiple nights (into tracks), to fit the
    those tracks that are consistent with an asteroid orbit,
    SSObjects
    , and to update
    SSObject
    the
    table. By its nature this process
    DIA-
    is
    Sources
    being associated and disassociated
    SSObjects
    . It is expected
    with that a frequency
    one day for these iterations
    SSObjects
    will(i.e.
    be update
    the each day) will be sufficient.
    3.5.1 Input Data
    DIAObject Database:
    Unassociated
    DIASources
    from the previous night of observing.
    means
    DIAObjects
    that were newly created during the previous night
    be associatedDIAObjects
    with known
    .
    DIASources
    associated
    SSObject
    with
    inan
    the night are
    still passed through the MOPS machinery
    SSObject Database:
    The catalog of known solar system sources
    Exposure Metadata:
    A description of the footprint of the observations
    tions of bright stars or a model for the detection threshold
    sky (including gaps between chips)
    The contents of this document are subject to configuration control by the LSST
    28

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    3.5.2 Output Data
    SSObject Database:
    An updated
    SSObject
    database
    SSObjects
    withboth added and pruned
    as the orbital fits are refined
    DIASource Database:
    A updated
    DIASource
    database
    DIASources
    withassigned and unas-
    signed
    SSObjects
    to
    Tracklet Database:
    A temporary database of tracklets measured durin
    will be persisted for at least a lunation.
    3.5.3 Tracklet identification
    From multiple visits withinDIASources
    a night,
    to form
    linktuples
    unassociated
    (or n-tuples)
    of
    DIASources
    Pipeline Tasks
    •Extract unassociated
    DIASources
    from the Level 1 database
    •Link
    DIASources
    into tracklets assuming a maximum velocity for
    maximum velocity will be based 19
    on
    ].
    a prior
    Foreachtrackleta
    as described
    ity vector will be calculated to enable pruning or merging
    a data set.
    •Merge tracklets by clustering in velocity and position
    time). Tracklets can contain multiple points and all
    will be stored. In the process
    DIASources
    of that
    merging
    aretracklets
    not a good fit
    for the merged tracklet will be remove and their associated
    tracklet database. Moving or trailed sources will incorporate
    source when linking. Details of
    DIASource
    the linkage
    implementation
    is described
    of
    in
    6.26
    The contents of this document are subject to configuration control by the LSST
    29

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    •Temporarily persist a database of tracklets. This database
    30 days of data but, depending on resources available,
    3.5.4 Precovery and merging of tracklets
    Tracklets are matched and
    SSObjects
    merged
    and
    with
    removed
    existing
    from the Tracklet
    database. This culls
    DIASources
    any that
    tracklets
    obviously
    or
    belong
    SSObject
    to an existing
    from the rest of the processing.
    Pipeline Tasks
    •Return all tracklets identified within a given night of
    •Return the footprints of each visit and the time of the
    •Extract
    SSObjects
    from the
    SSObject
    database and propagated those orbits
    tion and time of a visit. Details of this orbit propagation
    6.25
    •Merge (precovery) the tracklets
    SSObject
    trajectories
    with the projected
    SS-
    and refit
    Object
    orbit model.
    DIASources
    previously associated
    SSObject
    maywith
    no longer
    an
    fit
    the updated
    SSObject
    orbits.
    DIASources
    Thesewill be removed
    SSObject
    from
    and
    the
    returned as unassociated
    DIAObjects
    to the level 1 database. All tracklets
    these
    DIAObjects
    will be returned to the tracklet database. Details
    precovery are described
    6.27
    in
    3.5.5 Linking tracklets and orbit fitting
    Given a database of tracklets constructed from a window
    tracklets into tracks assuming a quadratic approximation
    with orbital models
    SSObject
    and database.
    update the
    Pipeline Tasks
    The contents of this document are subject to configuration control by the LSST
    30

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    •Extract all tracklets from the tracklet database for a
    days)
    •Merge tracklets into tracks based on their velocities
    are pruned by fitting a quadratic relation to the positions
    correction to the positions of the sources). Efficiency
    vided by a spatial index such
    6.28
    ). as a kd-tree (see
    •Fit an orbit to each candidate track using a tool such as
    (https://github.com/oorb/oorb) and,
    DIASources
    for poorly
    and
    fitting
    associated tracklets to their respective databases for
    •Merge
    SSObjects
    that have similar orbital parameters based on
    the six dimensional orbital parameter
    SSObjects
    willspace.
    need toMerged
    be refit
    and any poorly
    DIASources
    fitting
    (and associated tracklets) returned
    databases for subsequent reprocessing. Details
    6.29
    of this
    3.5.6 Global precovery
    For all new or
    SSObjects
    updated
    propagate the orbits to the positions
    servations of all tracklets
    DIAObjects
    to and
    “precover”
    orphan further support
    This will prune the number
    DIAObjects
    ofthat
    tracklets
    will require
    and
    merging in
    observations.
    Pipeline Tasks
    •Return all tracklets identified within a given night of
    •Return the footprints of each visit and the time of the
    •Extract orbits for all
    SSObjects
    new
    and
    orpropagate
    updated
    the positions to
    of the observations for all visits covering the extent
    days,6.25
    (see
    )
    •Merge the tracklets with
    SSObject
    the
    positions
    projected
    and
    SSObject
    refit
    orbit
    the
    model. Poorly
    DIASources
    fitting
    (and associated tracklets)
    SS-
    will be removed
    Object
    and returned as unassociated
    DIAObjects
    to the Level 1 database (as described
    in
    6.27
    ).
    The contents of this document are subject to configuration control by the LSST
    31

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    The process for precovery
    SSObject
    and updating
    models isof
    naturally
    the
    iterative
    the pruning of poorly
    DIAObjects
    and
    fitting
    tracklets).
    SSObjects
    Updates
    as part
    of the
    of
    each night of operations should enable sufficient iterations
    be rerun multiple times per day. The computationally expensive
    are the orbit propagation and the orbit fitting. Resources
    be reduced be removing the initial precovery stage but at
    of tracklets that would be available for matching into tracks.
    pre-calculated and modelled as polynomials to enable fast
    Extending the Global Precovery
    DIASources
    to(i.e.
    include
    onesingleton
    that are not merged
    into tracklets) would enable the identification of asteroids
    (where an object moves outside of the nightly survey footprint
    second visit is not obtained for a given field).
    3.5.7 Prototype Implementation
    Prototype MOPS
    codeshttps://github.com/lsst/mops_daymops
    are available at
    and
    https://
    github.com/lsst/mops_nightmops
    Current DayMOPS prototype already performs
    computational envelope envisioned for LSST Operations,
    required completeness requirement.
    The contents of this document are subject to configuration control by the LSST
    32

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Figure
    Detection
    6:
    and orbital modelling of moving sources within
    generation from revisits, filtering
    SSObjects
    of
    ,tracklets
    fitting of tracks
    based on
    and
    known
    orbits to tracklets, pruning
    DIAObjects
    based
    of tracklets
    on new
    and
    SSObjects
    updated
    .
    The contents of this document are subject to configuration control by the LSST
    33

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4 Calibration Products Production
    This section details the input data and algorithms required
    essary for the photometric calibration of the LSST survey.
    products is covered in other sections of this document. The
    given4.2
    in
    and
    § 4.5
    §, which define the source
    i.e.
    which
    of these
    willdata,
    be provided
    the camera team and which will be measured on
    4.3
    the
    and4.6
    mountain.
    §
    list the various output data products from the Calibration
    4.1 Key Requirements
    The work performed in this WBS serves several complementary
    •It will enable the production of calibration data products
    tometric Calibration
    LSE-180
    ) andPlan
    other
    ( planning documents. This
    characterization of the sensitivity of the LSST system
    the transmissivity of, and emission from, the atmosphere;
    •It will characterize detector anomalies in such a way
    by the instrument signature removal routines in the Single
    (WBS 02C.03.01) or, if appropriate, elsewhere in the system;
    •It will provide updated values of the crosstalk matrix
    (for DRP) for correction of the raw data;
    •It will allow for characterization of the optical ghosts
    4.2 Inputs
    The following section details the input datasets which will
    ucts Pipeline which will be acquired by the operations team
    of these will be acquired
    e.g.
    flats,
    frequently,
    while some will be acquired
    quently,
    e.g.
    the gain and linearity values. It should be noted
    and as such, the algorithmic sections for items that are
    are shown as “None” as these will have been previously developed.
    The contents of this document are subject to configuration control by the LSST
    34

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    items are re-listed
    outputs
    in
    ,
    section
    the
    where the algorithms to recalculate/monitor
    on the mountain are defined.
    4.2.1 Bias Frames
    A set of bias frames used for the production of the master
    many zero second exposures with the shutter remaining closed,
    cadence.
    • Algorithmic component: None - these just need to be taken.
    4.2.2 Gain Values
    Camera Team deliverable
    The gain values for all amplifiers
    /ADU;in
    note
    thethat
    camera,
    these
    inare required
    high accuracy (0.1%), as they are used in determination of
    • Algorithmic component: None.
    4.2.3 Linearity
    Camera Team deliverable
    The linearity curve for each amplifier in the camera, as well
    non-linearity curves should be considered unreliable.
    • Algorithmic component: None.
    4.2.4 Darks
    Sets of long dark
    300s)
    frames
    with(the actual exposure length optimized
    in the delivered sensors, the delivered read-noise, and
    integrated cosmic ray flux and radioisotope contamination.
    • Algorithmic component: None - these just need to be taken.
    The contents of this document are subject to configuration control by the LSST
    35

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4.2.5 Crosstalk
    Camera Team deliverable
    The crosstalk matrix for every pair of amplifiers in the camera.
    expected to be a very sparse.
    • Algorithmic component: None.
    4.2.6 Defect Map
    Camera Team deliverable
    A list of all bad (unusable) pixels in each CCD,
    i.e.
    ones
    as well as
    which should be flagged as such during processing.
    • Algorithmic component: None.
    4.2.7 Saturation levels
    Camera Team deliverable
    The lowest level (in electrons), for each amplifier, at which
    pixels. If necessary, they will also provide the
    i.e.
    if
    level at
    the serial saturates at a lower level than the parallels).
    • Algorithmic component: None.
    4.2.8 Broadband Flats
    Sets of flats taken through the standard LSST filters. Flats
    levels to measure the “brighter-fatter effect”coefficients
    of “superflats” - sets of high-flux
    㼀 㘀㄀
    , flats
    possibly
    㼀 ㈀㄀㄀
    with
    ). The
    many
    superflats
    repeats (
    taken for “brighter-fatter effect”characterization will
    effect is not expected to evolve with time.
    • Algorithmic component: None - these just need to be taken.
    The contents of this document are subject to configuration control by the LSST
    36

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4.2.9 Monochromatic Flats
    Sets of ‘monochromatic’
    c.
    1nm bandwidth
    (
    and spacing) flat-field screen
    no filter/glass in the beam.
    • Algorithmic component: None - these just need to be taken.
    4.2.10 CBP Data
    Sets of images taken with the Collimated Beam Projector (CBP).
    steps in these datasets are preliminary. All CBP data will
    LSST ISR, except without the application of flat-fielding.
    will then be used to measure the number of counts associated
    • Algorithmic component: Scripting the CBP/8.4m to take
    The scripting/control requirements for4.4
    the CBP are dealt
    CBP dataset
    Sets
    1
    of CBP images scanned in wavelength at 1nm resolution
    3
    every 1nm for
    a fixed set of spot positions on the camera, and for fixed footprint
    in the beam.
    CBP dataset
    Sets
    2
    of CBP images scanned in wavelength at 20nm
    while rotating the CBP about a pupil to move the spot pattern
    footprint on M1. No filter should be in the beam.
    CBP dataset
    Sets
    3
    of CBP images scanned in wavelength at 20nm
    for a fixed set of spot positions on the camera, and for a number
    minimum number of footprints
    c.
    6 for a 30cmis
    CBP, but in reality the use
    will be explored to test the assumption of azimuthal symmetry.
    beam.
    3
    1nm ‘resolution’ here denotes the bandwith of the light source, and
    should, however, be noted that the accuracy on the wavelength calibration
    0.1nm level.
    The contents of this document are subject to configuration control by the LSST
    37

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    CBP dataset
    Sets
    4
    of CBP images scanned in wavelength at 1nm resolution
    a fixed set of spot positions on the camera, and for a fixed
    every filter.
    N.b.
    the wavelength range for each scan need only cover
    filter transmits appreciable light.
    CBP dataset
    Sets
    5
    of CBP images scanned in wavelength at 20nm
    for a fixed set of spot positions on the camera, and for fixed
    every filter.
    CBP Crosstalk Measurement
    Sets of CBP images taken with a suitably-designed
    mask to allow identification and measurement of all ghost
    crosstalk. The simplest sparse mask would have only a single
    amplifier in the camera in turn (but less sparse solutions
    lengths used are unimportant, and there are no constraints
    choice. This will be particularly necessary should LSST
    for example for use with 30s integrations, as crosstalk coefficients
    4.2.11 Filter Transmission
    The transmission curves (transmission as a function of wavelength)
    tion of filter position. This is to be delivered by the filter
    but is input data which will not be measured by DM. The required
    in keeping with the resolution of the monochromatic flats.
    • Algorithmic component: None.
    4.2.12 Atmospheric Characterization
    These are the external measurements
    e.g.
    of
    the
    atmospheric
    barometricparameters,
    pres-
    sure, ozone and temperature, provided by measurement systems
    • Algorithmic component: Interfacing with the site team
    ment, to automate obtaining the measurements in a machine-readable
    ozone data from satellites.
    The contents of this document are subject to configuration control by the LSST
    38

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4.3 Outputs from the Calibration Product Pipelines
    Pipelines
    This section details the outputs from the Calibration Products
    production of each item are defined, and includes provision
    previously listed as “camera team deliverables”.
    4.3.1 Master Bias
    A trimmed, overscan subtracted, master bias frame from the
    ing the median of several-to-many bias frames for each CCD
    • Algorithmic component: Given LSST’s 2s readout, we do not
    rays explicitly; a robust stacking algorithm should be sufficient.
    rithm currently
    pipe_drivers
    exists
    . The final version must be configurable
    vector- or array-type overscan subtraction.
    4
    If there is significant structure
    regions or the bias images themselves, some summary of this
    data to ensure that the fixed-pattern in each observation
    4.3.2 Master Darks
    A trimmed, overscan and bias-frame subtracted, master dark
    cal plane. These are produced by taking the
    c.
    300s)
    median
    dark
    of several-to-many
    exposures, which are subsequently scaled to 1s exposure
    • Algorithmic component: The individual frames will be run
    ing (including cosmic ray removal) before being combined;
    using the standard LSST image stacking code, and a prototype
    rently exists
    pipe_drivers
    in
    . The final version must be configurable to
    or array-type overscan subtraction, and be robust to contamination
    coadding.
    4
    If the readout noise in any channel is too low (relative to the gain)
    a simple fix is to
    (
    e.g.
    add
    3)sets
    biasof
    exposures before creating the stacked image.
    The contents of this document are subject to configuration control by the LSST
    39

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4.3.3 Master Linearity
    Linearity curves for each amplifier
    4.2.3
    in
    , unless
    the camera;
    updated
    identical
    during
    operations.
    • Algorithmic component: An algorithm will need to be written
    from raw data, either from binned flats, CBP data or “ramp
    treatment, as the “brighter-fatter effect”can masquerade
    the algorithm developed by the Camera Team to supply the
    used to make this measurement to sufficient accuracy. The
    correction during ISR is currently being implemented by
    calculate these after bias subtraction, or be consistent
    during ISR.
    4.3.4 Master Fringe Frames
    Compound (polychromatic) fringe frames, dynamically created
    trum of the atmosphere at the time of observation, if necessary.
    night sky’s emission spectrum is sufficiently stable so as
    first few PCA components of the fringe pattern will be used
    • Algorithmic component: Construction of
    monochromatic
    these fringe frames
    flats
    , likely using the existing
    pipe_drivers
    PCA
    . algorithm in
    4.3.5 Master Gain Values
    The gain values for all amplifiers
    /ADU; identical
    in the camera,
    to § in
    4.2.2
    , unless updated
    during operations, though it is thought that this will likely
    • Algorithmic component: Whilst highly accurate initial
    input (to better than 0.1%), monitoring the evolution of
    currently an unsolved problem. The algorithm to determine
    tially tricky and will need to be developed.
    It will be possible to monitor the relative gain within a
    fields be continuous across amplifier boundaries; this is,
    boundaries.
    DM-6030
    Ticket
    exists to explore the possibility of using
    The contents of this document are subject to configuration control by the LSST
    40

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    unavoidable radioisotope contamination inside the camera
    5
    then
    another method will need to be devised. The necessary accuracy
    be firmly established.
    Two main techniques exist to measure
    photon
    the
    transfer
    gain
    technique
    in CCDs:
    curve
    the
    (PTC), and illumination of the sensor with
    㘀㘀
    Fe X-rays or those from another similar
    tope. Both of these techniques need to be applied with care
    the “brighter-fatter effect”, it is not clear to what accuracy
    gain, though sufficiently large binning of flat-fields can
    this effect, and while radioisotope gain measurement achieves
    illuminate the focal plane in a suitable manner is uncertain.
    㘀㘀
    Fe measurement
    technique be used, the flux measurement will use the standard
    measurement algorithms.
    4.3.6 Master Defects
    A list of all the bad pixels 4.2.6
    in ,
    each
    unless
    CCD;updated
    identical
    during
    to §operations.
    • Algorithmic component: Perform statistical analysis of
    exposures to derive an updated defect list. These algorithms
    Camera and electro-optical test teams.
    4.3.7 Saturation Levels
    The level (in electrons), for each amplifier, at which charge
    identical
    4.2.7
    ,to
    unless
    §
    updated during operations.
    • Algorithmic component: This will be measured using CBP
    levels could also be measured by saturating many stars in
    written to detect where saturation is occurring using the
    saturation levels.
    4.3.8 Crosstalk
    The crosstalk matrix element for every pair of
    4.3.8
    amplifiers
    ,
    unless updated during operations. The probability that
    5
    Merlin’s estimate is that the likelihood of failure is moderate-to-high,
    The contents of this document are subject to configuration control by the LSST
    41

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    as the validity of these values depends on them being measured
    configuration. This is due to the inter-CCD and inter-raft
    the capacitive couplings, which, though supposedly small
    raft coupling), depend on the exact physical locations of
    with respect to one another. It is therefore necessary to
    mountain using the CBPcrosstalk
    using the
    . dataset
    • Algorithmic component: In the un-multiplexed limit, this
    around the focal plane and measuring the positive and negative
    biguating these from optical ghosts using the fact that electronic
    coordinate offsets whereas optical ghosts will move as a function
    multiplexing will be possible using a multi-pinhole CBP
    to be determined, and depends on the final properties of the
    We baseline for a single spot mask, a one-spot-per-CCD mask,
    ideally a one-spot-per-amplifier mask.
    CBP dithering scripts will be written which will involve
    followed by either performing a camera rotation or by re-raster
    position for the previous focal plane positions to differentiate
    tical ghosts. Code to perform this differentiation will be
    coupling coefficients. Further reading on crosstalk
    23
    ].
    in LSST
    Confirmation of the measured crosstalk matrix will be performed
    urated stars’ bleed trails, or
    35
    ].
    cosmic rays in dark frames[
    4.3.9 Master Impure Broadband Flats
    A set of broadband master flats, one per filter, produced
    trimmed, bias-, overscan-, and dark-corrected flat-field
    include
    any ghosted or scattered light, and will be used to monitor
    etc.
    on the optics. A set of broadband flats will be acquired
    master flats, and if significant change is found, this will
    sary input data4.2
    products
    , and thein
    regeneration
    §
    of the corresponding
    • Algorithmic component: Construction
    pipe_drivers
    . algorithm exists
    The contents of this document are subject to configuration control by the LSST
    42

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4.3.10 Master Impure Monochromatic Flats
    A set of master flats produced by taking the median
    c.
    1nm) of a set
    trimmed, bias-, overscan-, and dark-corrected flat-field
    include
    any ghosted or scattered light.
    • Algorithmic component: Construction
    pipe_drivers
    . algorithm exists
    4.3.11 Master Pure Monochromatic Flats
    A set of master flats produced by taking the median
    c.
    1nm) of a set
    trimmed, bias-, overscan-, and dark-corrected flat-field
    exclude
    any ghosted or scattered light, with the ghost exclusion
    • Algorithmic component: Having performed a starflat-like
    6
    of the
    CBP data
    , and
    having normalized the results, we will fit a surface through
    for the whole camera. A spline would be a reasonable choice;
    splines,orathinplatespline. RHLwouldstartwiththeformer
    The dome-flat is then divided by this surface, giving an estimate
    chip correction. A curve is then fitted to this correction,
    This should be close to the values
    CBP data
    (and
    derived
    can preserve
    from thediscontinuities
    in the QE across chips which the fitted curves have a hard
    then iterated a few times, with each iteration resulting
    which we are therefore better able to model. This process
    wavelengths, chosen so that the variation of these corrections
    well captured. We will then know the relative QE for all the
    of wavelength, in the absencefiltertransmissioncurves
    of a filter. Then,
    theusing
    relative
    the
    QE for all the pixels in the camera for each filter can be determined
    is our monochromatic
    photometric
    LSST’s
    flatfield.
    plans for
    See
    Calibrated
    the
    for
    Photometry
    further reading.
    4.3.12 Master PhotoFlats
    A set of master flats, each composedpure
    of a monochromatic
    linear combination
    ,
    flats
    weighted by a flat-spectrum source (or other predefined standard
    6
    Some adaptation of the stack’s starflat processing code will likely
    data, but his code by-and-large already exists or is independently under
    The contents of this document are subject to configuration control by the LSST
    43

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    dard atmosphere, and observed through each filter. Each
    the median of many exposures. This will only be necessary
    being applied, though this product will always be used to
    ate sky-spectrum used for the weighting.
    • Algorithmic component:pure
    The monochromatic
    combination
    is simple,
    offlats though
    the “standard atmosphere” and “standard SED” remain to be
    4.3.13 Master Low-resolution narrow-band flats
    A set of master flats produced by taking the median of a set
    and wavelength)
    4.3.11
    version
    , used to
    ofsave
    §
    memory in the conversion of
    from the flattened data using the current sky colour to the
    • Algorithmic component: Scripting to perform the necessary
    and characterization of its output, as the pulse energy will
    4.3.14 Pixel Sizes
    A map of the (effective) pixel-size distortions.
    width
    At worst,
    뜀츉
    height
    뜀 ㌀
    dat-
    acube of floats. Pixel size distortions include small-scale
    stitching/tiling artifacts, tree-rings, and any other effects
    • Algorithmic component: The algorithm to measure this is
    problem. It has been claimed by Aaron Roodman, Michael Baumer
    that these can be measured from flat-fields, but the problem
    the stability (nay, validity?) of their measurements is
    Further thought is required to establish whether their method
    another one. It is not obvious how the problem can be made
    is ongoing in the DESC Sensor Anomalies Working Group (SAWG)
    might help inform future thinking on the matter.
    4.3.15 Brighter-Fatter Coefficients
    The coefficients needed to model the “brighter-fatter effect”.
    number of floats per CCD, but this is not yet entirely clear.
    late these will likely
    superflats
    be restricted
    various flux
    tolevels, with the possible
    of some star fields for verification of the coefficients.
    The contents of this document are subject to configuration control by the LSST
    44

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    • Algorithmic component: A number of techniques exist to
    by members of the Princeton LSST/HSC group). Code already
    efficients, and apply the corrections using a slightly enhanced
    technique.
    4.3.16 CTE Measurement
    Measurement of the charge transfer efficiency for each amplifier/column
    most simple case, where the dominant trap is close to the
    thus affects all columns equally, this would be a single number
    of complexity would be a number per column, with the still
    characterizing the specific defects and their locations on
    a per-pixel product, though this could be simplified with
    defect maps. The nominal case should likely be considered
    because if the number of columns with significant effects is
    likely just be masked out rather than corrected.
    • Algorithmic component: Measurement of CTE is subtle, though
    exist for doing so. Using the
    㘀㘀
    Fe method may not be possible due to the probable
    radioisotope source in
    extended
    the camera,
    pixel
    but(EPER)
    edge
    the
    response
    method and
    flat-field correlation method would both be possible using
    We expect to be able to reuse the measurement algorithms from
    have been ported to run within the DM framework.
    4.3.17 Filter Transmission
    Monitoring of filter
    in-situ
    transmission
    . As well as the filter transmission measurement
    vided by the camera team/vendor
    4.2.11
    , we further
    in §baseline the development
    dure for monitoring the filter response at 1 nm resolution
    ments with and without the filters in the beam, and averaging
    Whilst the flat-top portion of the filter pass-band will be
    gradient and minimal ringing, the transmission across the
    extinction or mirror degradation and its monitoring is therefore
    of the filter edges. The evolution of the edges of the filter
    the best of the ability of the photometric calibration hardware,
    The contents of this document are subject to configuration control by the LSST
    45

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    by the laser performance and ability to characterize its
    • Algorithmic component: Created
    4.2.10
    from measurements in §
    4.3.18 Ghost catalog
    A catalog of the optical ghosts and glints which is available
    Detailed characterization of ghosts in the LSST system will
    operational. The baseline design therefore calls for this
    from precursor instrumentation; we
    e.g.
    note
    HSCthat
    are well-known
    ghosts and
    and more significant than are expected in LSST.
    4.3.19 Spectral Standards
    A set of standard stars, spectrally characterized above
    colors, and lying within an appropriate magnitude range,
    pointing; the likely source of this data is Gaia. However,
    source
    7
    , a catalog will be carefully generated using the survey’s
    ing an übercal/jointcal type approach.
    • Algorithmic component: Color transformations need to
    surements, based on assumptions about
    i.e.
    not
    theusing
    objects’
    onlyintrinsic
    color
    terms. In the case that Gaia does not provide the catalog,
    Global Calibration Model (FGCM) implemented by Eli Rykoff
    be used. The latter process would likely be able to share
    the reductions performed for the auxiliary/Calpyso/calibration
    4.3.20 Spectrophotometric Standards
    A set of photometrically characterized stars with well known
    sky. This will likely be comprised of DA white-dwarves,
    i.e.
    fainter) set of stars which will be bootstrapped from
    21
    ].
    the
    • Algorithmic component: Exactly how these stars will be
    7
    This is likely to be the case in the
    -band, where Gaia’s SNR for the BP spectra falls off
    The contents of this document are subject to configuration control by the LSST
    46

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    4.3.21 Astrometric Standards
    A set of stars used
    absolute
    astrometric
    for the
    calibration
    i.e.
    the determination
    of each visit,
    of the nominal pointing for each exposure. The likely source
    4 magnitudes of overlap between Gaia’s faintest astrometric
    saturated sources, with the absolute astrometry provided
    to be
    450
    as at the faint end. This data will be made available
    4.4 CBP Control
    The procurement of the CBP hardware includes that of the necessary
    s/software. T&S TCS own the task of taking the vendor-provided
    these into real-world usable routines by constructing
    e.g.
    homing,
    higher
    slew-to-position,
    etc.
    ,mount
    though
    a mask
    it should be noted that this is
    and purely illustrative list of example functions, and not
    that will be provided. T&S will also provide a pointing model
    Control scripts for the CBP and interfaces with the OCS will
    desired measurements, especially as several, if not all
    with the 8.4m. As well as writing the necessary scripts to
    lined
    4.2
    in
    , it
    § will also be necessary to deliver a coordinate
    the CBP to maintain a fixed position on the focal plane whilst
    of the pupil, and vice versa.
    4.5 Calibration Telescope Input Calibration Data
    This section details the input data required to calibrate
    scope itself. Broadly, this will include
    4.2
    , but
    most
    namely:
    of the ingredients
    •Gain values
    Less accuracy is needed here than for the main camera;
    using flats will likely be sufficient if the data is binned
    for the “brighter-fatter effect”, and will therefore
    the 8.4m.
    The contents of this document are subject to configuration control by the LSST
    47

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    •Crosstalk matrix
    This will reuse the algorithm designed for the 8.4m assuming
    version available, otherwise this will be calculated
    rays.
    •Linearity curves for each amplifier
    This will reuse the algorithm designed for the 8.4m.
    •Defect map
    This will reuse the algorithm designed for the 8.4m.
    •Saturation levels
    This will reuse the algorithm designed for the 8.4m
    CBP version available, otherwise it will involved the
    •Bias frames
    This will reuse the algorithm designed for the 8.4m.
    •Dark frames
    This will reuse the algorithm designed for the 8.4m.
    •Broadbandflat-fields
    This will reuse the algorithm designed for the 8.4m.
    •Monochromaticflat-fields.
    8
    This will reuse the algorithm designed for the 8.4m.
    •Disperser (grating/grism) transmission
    The baseline specification is for a Ronchi grating to
    in the optical design. Although the transmission of a
    because it will be placed in a non-parallel beam second
    that its effective transmission will not be perfectly flat,
    for. Furthermore, a grism or blazed-grating are also
    dispersive element, neither of which have flat responses
    smoothly varying nature of their transmission functions
    at the same time as performing the fit to the atmospheric
    8
    It is confirmed to be part of the baseline design that there will be both
    sources at the auxiliary/Calpyso/calibration telescope.
    The contents of this document are subject to configuration control by the LSST
    48

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Further to these standard camera calibration data products,
    will also be required, which will either be derived from
    final CBP prototype for direct measurement.
    4.6 Calibration Telescope Output Data
    This section details the calibrated outputs from the auxiliary/Calpyso/calibration
    which, like items
    4.3
    ,in
    are
    section
    outputs
    § from the Calibration Products
    used during photometric calibration at various levels.
    4.6.1 Atmospheric Absorption
    As shown in7,Figure
    the determination of the atmospheric transmission
    ages, one dispersed, and one direct and unfiltered, acquired
    Calpyso/calibration telescope, where the camera rotator
    trum along the parallactic angle. Both images are initially
    as per the normal image processing, with cosmic rays detected
    with defective pixels.
    Both images then have their PSFs measured, and are astrometrically
    way; the direct image is treated exactly as normal, while
    brightest objects in the image will be used, thereby preventing
    detections due to the spectra.
    The direct image is suitably flux-scaled and warped, and is
    image to remove the zeroth-order light, leaving only the
    first order at least, this will remove the sky-background
    thermore, as the stars used for atmospheric characterization
    sky-background is thought to be a negligible contribution.
    Using the astrometry and the nominal dispersion relation
    the regions in which the spectra fall are identified and the
    spectral lines in these crude uncorrected-spectra are identified
    proved estimate of the dispersion relation. This is performed
    found in
    촉 㸀 ㈀
    the
    and
    촉 㸀 缄㈀
    spectra. This first approximation of the
    The contents of this document are subject to configuration control by the LSST
    49

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    is used to calculate the incident wavelength as function
    then used to construct an appropriate flat-field for the main
    around the source, as a set of parallel stripes perpendicular
    structed from the monochromatic flatfield data-cube. If the
    CCD’s serial or parallel
    i.e.
    if we choose
    direction,
    to disperse along the parallactic
    above, then the Bresenham algorithm will be used to construct
    Having applied the flatfield, the 1D spectra are then extracted
    sian profile derived from the initially measured
    correct
    model
    PSF. Whilst
    to fit here, the fitting is often not stable, and this is thought
    wings, which is suppressed by using a Gaussian, while the
    level. The extracted spectrum is then flux calibrated and
    tamination
    9
    . A more precise wavelength calibration is then performed
    in this corrected spectrum, taking into account the effect
    resulting in a spectrophotometrically calibrated measurement.
    The
    source’s true
    and the
    SEDcalibrated spectrophotometric observation
    conjunction with the observational
    e.g.
    the zenith
    meta-data,
    angle, temperature,
    metric pressure, to derive an empirical measurement of the
    absorption profile is then fitted to an atmospheric transmission
    ered spectral absorption measurement, as well to provide
    state of the atmosphere at the time of observation.
    4.6.2 Night Sky Spectrum
    The acquisition of a night sky spectrograph is unlikely as
    ification. However, in the eventuality that such an instrum
    the determination of the emission spectrum of the night
    bration telescope
    뤉 ꤄ ㌀㄀㄀
    boresight,
    10
    , which will
    withbe used to synthesize
    matching the sky’s
    monochromatic
    SED using the.dome flats
    • Algorithmic component: Assuming we have a sky spectrograph
    of a sky spectrograph,
    뤉 ꤄ ㈀㄀
    spectrum
    an will be acquired using standard/narrowband
    Furthermore, if the fringe structures
    i.e.
    they arewell
    sufficiently
    described
    3
    stable,
    9
    It should be noted that strictly speaking, second order light contamination
    described above. If the effect is small, a simple QE curve will likely
    iterative approach to the flat-fielding will be taken.
    10
    It is not entirely clear yet whether these will be taken on the Calypso
    The contents of this document are subject to configuration control by the LSST
    50

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Figure
    Flowchart
    7:
    depicting the atmospheric absorption measurement
    .
    The contents of this document are subject to configuration control by the LSST
    51

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    PCA components, we may be able to simply use a classic fringe
    4.7 Photometric calibration walk-through
    The Calibration Products Production section aims to provide
    photometrically calibrate the entire LSST survey, visit-by-visit
    at everything except a single photometric zero-point for
    The effective end-to-end instrumental throughput, as a function
    position, and time,ghost-corrected
    is known from the monochromatic
    and the
    filterflat-fields
    transmission
    . functions
    The atmospheric transmission as a function of wavelength,
    made, is known at some position in the field-of-view
    spectrophotomet-
    by taking
    ricly calibratedof
    stellar
    bright
    th
    缄 ㈀㄀
    spectrum
    th
    (magnitude) star, as measured
    auxiliary/Calpyso/calibration telescope, to the BP/RP
    sphere by Gaia.
    It should be noted that the effective delivered
    Gaia spectropho-
    spectral resolution
    tometry
    and the
    atmospheric absorption
    can be improved using model fits. The
    from Gaia,
    뤉 ꤄ 㔀㄀ 缄 㠀㄀
    with)(for the BP and RP spectra
    32
    ], can
    respectively[
    be fitted to stan-
    dard stellar spectral types, as will be done2].
    internally
    For the
    by
    atmospheric absorption profile,
    4.6.1
    , theas
    absorption
    describedfeatures
    in §
    from
    mosphere will be fitted to an atmospheric
    e.g.
    MODTRAN
    transmission
    etc.
    ), allowingmodel
    us to improve the delivered measurement of the spectral absorption
    time of observation.
    Images are initially flat-fielded using the color of the sky
    ensures that the sky background is correctly flat-fielded,
    tracted across amplifier and chip boundaries without residual
    is then reversed, and the resulting sky-background-subtracted
    pre-selected SED
    11
    in order to obtain a first-order estimate of the
    cessing, when an assumed SED has been derived for each object,
    made to adjust both for the derived SED and for the atmospheric
    11
    Whether this is a flat SED or some nominal SED
    e.g.
    a G-star’s remains TBD.
    The contents of this document are subject to configuration control by the LSST
    52

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    of observation. With each object now flat-fielded with the
    mospheric transmission and system response functions corrected
    metric zero-point for the
    photometric
    visit is fitted
    standard
    using
    . Thistherefore
    star set
    leaves just the overall flux level unknown, thus bringing
    point for the entire survey.
    This whole-survey zero-point can then calculated using
    to the definition of the Jansky, and two proposals exist for
    surements in conjunction with NIST calibrated photodiodes
    measure the absolute instrumental sensitivity, though
    pupil. The second is to use a ‘son-of-StarDICE’ type approach,
    stabilized LEDs of known wavelength and luminosity are observed
    iliary/Calpyso/calibration telescope, as their observations
    the absolute system response to be measured using observations
    pupil at a set of wavelengths.
    It should be noted that it is not yet known whether the atmospheric
    significantly across LSST’s field of view, and that this is
    time by wide-field cameras such as DECam and HSC. Should it
    transmission varies on spatio-temporal scales relevant
    further per-visit corrections by measuring the variation
    classification for all Gaia sources across the field of view.
    essary, measuring this variation anyway will allow the
    transmission to be constrained, providing a convenient quality-assurance
    this choice.
    4.8 Prototype Implementation
    While parts of the Calibration Products Pipeline have been
    tion Group LSE-180
    (seefor
    thediscussion), these have not been written
    Management software framework or coding standards. We therefore
    know-how, and rewrite the implementation.
    The contents of this document are subject to configuration control by the LSST
    53

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    5 Data Release Production
    A Data Release Production is run every year (twice in the
    a set of catalog and image data products derived from all
    the survey to the point the production began. This includes
    image analysis run in Alert Production, in addition to direct
    and coadded images. The data products produced by a Data Release
    rized in
    2.table
    Name
    Availability
    Description
    Source
    Stored Measurements from direct analysis of individual
    DIASource Stored Measurements from difference image analysis
    posures.
    Object
    Stored Measurements for a single astrophysical object,
    all available information, including coadd
    multaneous multi-epoch fitting, and forced
    not include solar system objects.
    DIAObject Stored Aggregate quantities computing by associating
    cated DIASources.
    ForcedSource
    Stored Flux measurements on each direct and difference
    position of every Object.
    SSObject
    Stored Solar system objects derived by associating
    inferring their orbits.
    CalExp
    Regenerated
    Calibrated exposure images for each CCD/visit
    snaps).
    DiffExp
    Regenerated
    Difference between CalExp and PSF-matched
    DeepCoadd Stored Coadd image with a reasonable combination
    olution.
    ShortPeriodCoadd
    Renegerated
    Coadd image that cover only a limited range
    BestSeeingCoadd
    Stored Coadd image built from only the best-seeing
    PSFMatchedCoadd
    Regenerated
    Coadd image with a constant, predetermined
    TemplateCoadd
    Stored Coadd image used for difference imaging.
    Table
    2:
    of public data products produced during a Data Release
    description of these data products can be found in the Data
    [LSE-163
    ].
    From a conceptual standpoint, data release production can
    executed in approximately the following order:
    1.We characterize and calibrate each exposure, estimating
    ground models, and astrometric and photometric calibration
    between processing individual exposures independently
    rived from multiple overlapping exposures. These steps
    The contents of this document are subject to configuration control by the LSST
    54

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Figure
    Summary
    8:
    of the Data Release Production image processing
    split into multiple pipelines, which are conceptually organized
    in sections
    5.1
    -5.5A final pipeline group discussed
    5.6
    simply operates
    in section the
    catalogs and is not shown here.
    The contents of this document are subject to configuration control by the LSST
    55

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    tion
    5.1
    2.We alternately combine images and subtract them, using
    time-variable sources while building coadds that produce
    Coaddition and image differencing
    5.2 is described in section
    3.We process coadds to generate preliminary object catalogs,
    ing, and the first phase of measurement.
    5.3This is discussed
    4.We resolve overlap regions in our tiling of the sky, in
    detected and processed multiple times.
    5.4 This is described
    5.We perform more precise measurements of objects by fitting
    ages, either simultaneously or individually,
    5.5
    as discussed
    6.After all image processing is complete, we run additional
    additional object properties. Unlike previous stages,
    on the sky, as it may use statistics computed from the
    characterization of individual objects.
    8, but
    This
    postpro-
    stage is
    cessing pipelines are5.6
    described in section
    This conceptual ordering is an oversimplification of the
    shown in Figures
    8and
    9, the first two groups are interleaved.
    Each pipeline in this the diagram represents a particular
    specific unit of data, but pipelines may contain additional
    to further subdivide that data unit. The processing flow
    tion between pipelines, indicated by cycles in the diagram.
    cycle will be determined (via tests on smaller productions)
    allowing us to remove these cycles simply by duplicating
    times. Decisions on the number of iterations must be backed
    release production processing can thus be described as a
    executed by the orchestration middleware, with pipelines
    as vertices. Most of the graph will be generated by applications
    begins, using a format and/or API defined by the orchestration
    parts of the graph must be generated on-the-fly;
    5.5.1
    this will
    The contents of this document are subject to configuration control by the LSST
    56

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Figure
    Data
    9:
    flow diagram for the Data Release Production image
    differencing pipelines. Processing proceeds roughly counterclockwise,
    per right with pipelines
    5.1
    described
    Each update
    in Section
    to a component of the central
    CalExp dataset can in theory trigger another iteration of a
    will “unroll” these loops before production begins, yielding
    of incrementally updated CalExp datasets. The nature of this
    iterations will be determined by future algorithmic research.
    described more fully in the text.
    The contents of this document are subject to configuration control by the LSST
    57

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    5.1 Image Characterization and Calibration
    The first steps in a Data Release Production characterize
    sures, by iterating between pixel-level processing of individual
    acterization” steps) and joint fitting of all catalogs overlapping
    ibration” steps). All ImChar steps involve fitting the PSF
    ually improving these as we iterate), while JointCal steps
    12
    ) and
    photometric solutions while building new reference catalogs
    is necessary for a few reasons:
    •The PSF and WCS must have a consistent definition of object
    tions from a reference catalog are transformed via the
    used to build the PSF model, but the PSF model is then used
    troids that feed the WCS fitting.
    •The later stages of photometric calibration and PSF modeling
    tion and colors to infer their SEDs. Magnitude and morphological
    ImChar stages that supersede those in the reference catalogs
    to update it in the subsequent JointCal stage, allowing
    be used for PSF modeling in the following ImChar stage.
    The ImChar and JointCal iteration is itself interleaved
    ence imaging, as described
    5.2Thisin
    allows
    section
    the better backgrounds
    be defined by comparisons between images before the final
    characterizations, and calibrations.
    Each ImChar pipeline runs on a single visit, and each JointCal
    all visits within a single tract, allowing tracts to be run
    overlap multiples tracts, however, and will hence be processed
    The final output data products of the ImChar/JointCal iteration
    CalExp (calibrated exposure)
    Exposure
    images.
    , and hence
    CalExp
    hasis
    multiple
    an
    compo-
    nents that we will track separately.
    12
    This is not limited to FITS standard transformations; see Section
    7.11
    The contents of this document are subject to configuration control by the LSST
    58

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    5.1.1 BootstrapImChar
    The BootstrapImChar pipeline is the first thing run on each
    It has the difficult task of bootstrapping multiple quantities
    etc.) that each normally require all of the others to be specified
    while the algorithmic components to be run in this pipeline
    and specific requirements are not; algorithms that are run
    algorithms that are run later, and some iteration will almost
    A plausible (but by no means certain) high-level algorithm
    pseudocode. Highlighted terms are described in more detail
    def
    BootstrapImChar(
    raw
    ,reference
    ,calibrations
    ):
    # Somdeatparoducts
    components
    arevisit
    -widaendsomaerepe-rCC;D
    # these
    imaginary
    dattaypes
    letussdeawlitbhoth
    .
    # VisitExposure
    alshoascomponents
    ; mosatreself
    -explanatory
    , and
    # {mi
    } =={image
    ,mask
    ,variance
    } (for"MaskedImage
    ").
    calexp = VisitExposure()
    sources = VisitCatalog()
    snaps = VisitMaskedImageList()
    # holds
    botshnaps
    , butonl{yimage
    ,mask
    ,variance
    }
    parallel
    for
    ccd
    in
    ALL_SENSORS:
    snaps[ccd]
    RunISR
    (raw[ccd])
    = [
    for
    sna
    i
    p
    n
    SNAP_NUMBERS]
    snaps[ccd].mask
    SubtractSnaps
    =(snaps[ccd])
    calexp[ccd].mi
    CombineSnaps
    =(snaps[ccd])
    calexp.psf
    FitWavefront
    = (calexp[WAVEFRONT_SENSORS ].mi)
    calexp .{image ,mask ,variance ,background}
    =SubtractBackground
    (calexp.mi)
    parallel
    for
    ccd
    in
    ALL_SENSORS:
    sources[ccd]
    DetectSources
    = (calexp.{mi,psf})
    sources[ccd]
    DeblendSources
    = (sources[ccd], calexp.{mi,psf})
    sources[ccd]
    MeasureSources
    = (sources[ccd], calexp.{mi,psf})
    matches
    MatchSemiBlind
    =
    (sources , reference)
    while
    converged:
    not
    SelectStars
    (matches , exposures)
    calexp.wcs
    FitWCS
    (=matches , sources , reference )
    calexp.psf
    FitPSF
    (matches
    =
    , sources , calexp .{mi ,wcs})
    WriteDiagnostics
    (snaps , calexp , sources)
    parallel
    for
    ccd
    in
    ALL_SENSORS:
    snaps[ccd]
    SubtractSnaps
    =
    (snaps[ccd], calexp[ccd].psf)
    calexp[ccd].mi
    CombineSnaps
    =(snaps[ccd])
    calexp[ccd].mi
    SubtractStars
    =(calexp[ccd].{mi,psf}, sources[ccd])
    calexp.{mi,background}
    SubtractBackground
    (calexp.mi)
    =
    parallel
    for
    ccd
    in
    ALL_SENSORS:
    sources[ccd]
    DetectSources
    = (calexp.{mi,psf})
    calexp[ccd].mi, sources[ccd] =
    ReinsertStars
    (calexp[ccd].{mi,psf}, sources[ccd])
    sources[ccd]
    DeblendSources
    = (sources[ccd], calexp.{mi,psf})
    The contents of this document are subject to configuration control by the LSST
    59

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    sources[ccd]
    MeasureSources
    = (sources[ccd], calexp.{mi,psf})
    matches
    MatchNonBlind
    =
    (sources , reference)
    calexp.psf.apcorr
    FitApCorr
    (matches
    =
    , sources)
    parallel
    for
    ccd
    in
    SCIENCE_SENSORS :
    sources[ccd]
    ApplyApCorr
    =(sources[ccd], calexp.psf)
    return
    calexp , sources
    Much of this pipeline is an iteration that incrementally
    proving the PSF model. This loop is probably only necessary
    be necessary to subtract brighter stars in order to detect
    latitude visits to require only a single iteration. The
    changes in behavior between iterations will be determined
    is also likely that some of the steps within the loop may be
    they depend only weakly on quantities that change between
    Input Data Product:
    Raw amplifier
    Raw
    images from science and wavefront
    across one or more snaps. Needed telescope telemetry (seeing
    ing) is assumed to be included in the raw image metadata.
    Input Data Product:
    A full-sky
    Reference
    catalog of reference stars
    ternal (e.g. Gaia) and LSST data.
    The
    StandardJointCal
    pipeline will later define a deeper reference
    one and the new data being processed, but the origin and depth
    alog is largely TBD. It will almost certainly include Gaia
    from other telescopes, LSST special programs, LSST commissioning
    last LSST data release. Decisions will require some combination
    commissioning team, specification of the special programs,
    curately type faint stars using the Gaia catalog, and policy
    the degree to which data releases are required to be independent.
    selected, it could also require a major separate processing
    the data release production pipelines.
    The contents of this document are subject to configuration control by the LSST
    60

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Input Data Product: Calibrations
    frames andCalibration
    metadata from
    Products .
    Pipeline
    This may include any of the data4.3
    products
    , though listed
    some will probably not be used until later stages of the production.
    Output Data Product:
    A preliminary
    Source
    version of the Source table.
    tain all of the
    columns
    DPDD
    Source
    in the
    schema
    MeasureSources
    if the is appropriately
    configured, but some of these columns are likely unnecessary
    data productStandardJointCal
    that feeds
    , and it is
    likely
    DPDD
    that
    columns
    otherwill
    non-be
    present for that role.
    BootstrapImChar also has the capability to produce even
    ble for diagnostic
    WriteDiagnostics
    purposes
    ).(see
    These tables are not associated
    photometric calibration or aperture correction, and some
    besides centroids, and hence are never substitutable for
    Output Data Product:
    A preliminary
    CalExp
    version of the CalExp (calibrated
    sure). CalExp
    Exposure
    is
    object,
    an
    and hence it has several components;
    creates the first versions of all of these components (though
    merely copied
    raw
    from
    images).
    the
    Some CalExp components are determined
    of a full FoV and hence should probably be persisted at the
    Background), while others are straightforward CCD-level
    tainty).
    RunISR
    DelegateISR
    to the
    algorithmic
    tocomponent
    perform standard detrending
    as brighter-fatter correction and interpolation for pixel-area
    these corrections will require a PSF model, and hence must
    a later stage when an improved PSF model is available.
    We assume that the applied flat field is appropriate for background
    SubtractSnaps
    DelegateSnap
    to the
    Subtraction algorithmic
    to mask artifacts
    component
    in the difference between snaps. If passed a PSF (as in the iterative
    The contents of this document are subject to configuration control by the LSST
    61

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    also interpolate them Artifact
    by delegating
    Interpolation
    algorithmic
    to the
    component.
    We assume here that the PSF modeled on the combination of the
    terpolation on the Snaps individually; if this is not true,
    Snaps when an artifact appears on either of them (or we could
    but that’s a lot more work for very little gain).
    CombineSnaps
    DelegateImage
    to theCoaddition algorithmic
    to combine
    component
    the
    two Snaps while handling masks appropriately.
    We assume there is no warping involved in combining snaps.
    instead consider treating each snap as a completely separate
    FitWavefront
    DelegateWavefront
    to the
    Sensor PSF algorithmic
    to generate
    component
    an approximate PSF using only data from the wavefront sensors
    (e.g. reported seeing). Note that we expect this algorithmic
    LSST Systems Engineering, not Data Management. We start
    wavefront sensors only because these should be able to use
    the science exposures, mitigating the effect of crowding;
    be unnecessary.
    The required quality of this PSF estimate is TBD; setting
    running a version of BootstrapImChar with at least mature
    gorithms on precursor data taken in crowded fields, and final
    ceessing full LSST camera data in crowded fields. However,
    and crowding is much more important than accuracy; this
    enough result for subsequent stages to prcoeed.
    SubtractBackground
    DelegateSingle
    to the Visit Background
    algorithmic
    Estimation
    com-
    ponent to model and subtract the background consistently
    The multiple backgrounds subtracted in BootstrapImChar
    we may or may not add the previous background back in before
    The contents of this document are subject to configuration control by the LSST
    62

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    DetectSources
    Delegate
    SourceDetectionalgorithmiccomponent
    to the
    tofindabove-threshold
    regions
    Footprints
    ( ) and peaks within them in a PSF-correlated version
    first detect on the original image (i.e. without PSF correlation)
    peak identification for bright blended objects.
    In crowded fields, each iteration of detection will decrease
    ber of objects detected. Because this will treat fluctuations
    tected objects as noise, we may need to extend PSF-correlation
    an image with correlated noise and characterize the noise
    DeblendSources
    DelegateSingle
    to the Frame Deblending algorithmic
    to split
    Footprints
    with multiple peaks into deblend
    HeavyFootprints
    families,
    that split
    and generate
    each pixel’s values amongst the objects that contribute
    MeasureSources
    DelegateSingle
    to the Frame Measurement algorithmic
    to
    measure source properties.
    In BootstrapImChar, we anticipate
    Neighbor Noise
    using
    Replacement
    approach
    the
    to de-
    blending, with the following plugin algorithms:
    •Centroids
    •Second-MomentShapes
    •Pixel Flag Aggregation
    •AperturePhotometry
    •Static Point Source Model Photometry
    These measurements will not be included in the final Source
    algorithms necessary to feed later steps (and we may not measure
    The contents of this document are subject to configuration control by the LSST
    63

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    MatchSemiBlind
    DelegateSingle
    to the Visit Reference Matching algorithmic
    to match source catalogs to a global reference catalog. This
    ensuring robust matching even when some CCDs have no matchable
    flux limits, or artifacts.
    “Semi-Blind” refers to the fact that the WCS is not yet well
    vided by the observatory), so the matching algorithm must
    offset between the WCS-predicted sources positions and the
    SelectStars
    Use reference catalog classifications and source
    stars to use for later stages.
    If we decide not to rely on a pre-existing reference catalog
    and other objects, we will need a new algorithmic component
    measurements.
    FitWCS
    DelegateSingle
    to theVisit Astrometric Fit
    to determine
    algorithmic
    thecomponent
    WCS of the image.
    We assume this works by fitting a simple mapping from the visit’s
    tem to the sky and composing it with the (presumed fixed) mapping
    and focal plane coordinates. This fit will be improved in later
    be exact;
    0.05 arcsecond accuracy should be sufficient.
    As we iterate in crowded fields, the number of degrees of
    allowed to slowly increase.
    FitPSF
    DelegateFull
    to the
    Visit PSF Modeling algorithmic
    to construct
    component
    an im-
    proved PSF model for the image.
    Because we are relying on a reference catalog to select stars,
    from the reference catalog to estimate SEDs and include
    If we do not use a reference catalog early in BootstrapImChar,
    The contents of this document are subject to configuration control by the LSST
    64

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    be wavelength-dependent. In either case the PSF model will
    pipelines.
    PSF estimation at this stage must include some effort to model
    if this is tracked and constrained separately from the model
    aspect of PSF modeling is considerably less developed, and
    research.
    As we iterate in crowded fields, the number of degrees of freedom
    be allowed to slowly increase.
    WriteDiagnostics
    If desired, the current
    source
    ,
    calexp
    state
    , and
    snaps
    of
    variables
    the
    may
    be persisted here for diagnostic purposes.
    SubtractStars
    all detected stars above a flux limit from
    model (including the wings). In crowded fields,
    SubtractBack-
    this should
    ground
    and
    DetectSources
    steps to push fainter by removing the brightest
    Sources classified as extended are never subtracted.
    ReinsertStars
    Add stars removed
    SubtractStars
    in
    back into the image, and merge
    sponding
    Footprints
    and peaks into the source catalog. Information
    detections will be propagated through the peaks.
    MatchNonBlind
    a single-CCD source catalog to a global reference
    by delegating
    the same
    to matching algorithm used
    . A separate
    in JointCal
    algorithm
    pipelines
    component may be needed for efficiency or code maintenance
    iting case of the multi-way JointCal matching problem that
    simpler implementation.
    “Non-Blind” refers to the fact that the WCS is now known well
    cant offset between WCS-projected source positions and reference
    The contents of this document are subject to configuration control by the LSST
    65

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    FitApCorr
    DelegateAperture
    to the
    Correction algorithmic
    to construct
    component
    a curve
    of growth from aperture photometry measurements and build
    other fluxes (essentially all flux measurements aside from
    predicted integrated flux at infinity.
    Additional research may be required to determine the best
    galaxy fluxes. Our baseline approach is to apply the same correction
    to stars, which is correct for small galaxies and defines a
    is formally incorrect for large galaxies, but there is (to
    approach.
    ApplyApCorr
    Delegate
    Aperture
    to the
    Correction algorithmic
    to apply aperture
    component
    corrections to flux measurements.
    5.1.2 StandardJointCal
    In StandardJointCal, we jointly process all of
    Boot-
    the Source
    strapImChar
    on each visit in a tract. There are four steps:
    1.We match all sources and the reference
    JointCalMatching
    catalog
    . This
    by delegating
    is a non-blind search; we assume
    BootstrapImChar
    the WCSs
    areoutput
    good enough
    by
    that we don’t need to fit for any additional offsets between
    matches will not include a reference object, as the sources
    deeper than the reference catalog.
    2.We classify matches to select a clean samples
    Joint-
    of stars
    CalClassification
    . The samples for photometric and astrometric
    ferent (for instance, we may require low variability
    proper motion only in the astrometric fit). This uses morphological
    information from source measurements as well as reference
    available). This step also assigns an inferred SED to
    this supersedes SEDs or colors in the reference catalog
    absolute calibration.
    3.We fit simultaneously for an improved astrometric solution
    a match to have the same position,
    Jointdelegating
    Astrometric
    algorithmic
    toFit
    the
    The contents of this document are subject to configuration control by the LSST
    66

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    component. This will need to correct (perhaps approximately)
    DCR, proper motion, and parallax; if it does not, it must
    (perhaps via outlier rejection). This requires that
    VisitInfo component of each CalExp, in order to calcluate
    ters to fit must be determined by experimentation on real
    number of degrees of freedom in the as-built system on different
    the algorithm must be flexible enough to fit a wide variety
    the WCS component for each CalExp.
    4.We fit simultaneously for a per-visit zeropoint and a smooth
    correction by requiring each star in a match to have the
    poch smoothed monochromatic flat fields produced by the
    delegating
    Joint
    to the
    Photometric
    algorithmic
    Fit component. This fit should
    the ability to fit per-CCD photometric zeropoints for
    small chance this fit will also be used to further constrain
    fields. This fit updates the PhotoCalib component for each
    In addition to updating the CalExp, WCS, and PhotoCalib,
    Reference dataset containing the joint-fit centroids and
    well as their classifications and inferred SEDs. The sources
    will be a securely-classified bright subset of the full source
    StandardJointCal may
    RefineImChar
    be iterated
    to ensure
    with
    the PSF and WCS converge
    the same centroid definitions. StandardJointCal
    BootstrapIm-
    is always
    Char
    , but
    RefineImChar
    orStandardJointCal
    may be the last step in the iteration
    proceding
    WarpAndPsfMatch
    with
    .
    If the Gaia catalog cannot be used to tie together the photometric
    ent tracts, a larger-scale multi-tract
    Global
    photometric
    Photometric
    fit must
    Calibration
    ), which would upgrade this step from a tract-level
    point. It is unlikely this sequence point would extend to
    once, but may happen in either
    FinalJointCal
    StandardJointCal
    . If the Gaia catalog
    or
    is
    cient for large-scale photometric
    Global Photometric
    calibration,
    may instead
    Fitting
    be run
    after the data release production as complete as a form of
    Before LSST’s atmospheric monitoring telescope, the Gaia
    The contents of this document are subject to configuration control by the LSST
    67

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    matic flats are available, photometric calibration will be
    pipeline commissioning (on both precursor data and some
    quire a more sophisticated
    Interim
    global
    Wavelength-Dependent
    fit (see
    )
    Photometric
    that uses multiple observations of stars to infer their
    transmission of the system as well as their magnitudes and
    transmission.
    5.1.3 RefineImChar
    RefineImChar performs an incremental improvement
    Boot-
    on the
    strapImChar
    , then uses this to produce improved source measurements,
    proved reference catalog, WCS, StandardJointCal
    PhotoCalib
    . Its steps
    produced
    are
    thus a strict subset
    BootstrapImChar
    of those
    . A pseudocode
    in
    description of RefineImChar
    is given below, but all steps refer
    5.1.1
    to
    : back to the descriptions
    def
    RefineImChar(
    calexp
    ,sources
    ,reference
    ):
    matches
    MatchNonBlind
    =
    (sources , reference)
    SelectStars
    (matches , exposures)
    calexp.psf
    FitPSF
    (matches
    =
    , sources , calexp .{mi ,wcs})
    parallel
    for
    ccd
    in
    SCIENCE_SENSORS :
    calexp[ccd].mi
    SubtractStars
    =(calexp[ccd].{mi,psf}, sources[ccd])
    calexp.{mi,background}
    SubtractBackground
    (calexp.mi)
    =
    parallel
    for
    ccd
    in
    SCIENCE_SENSORS :
    sources[ccd]
    DetectSources
    = (calexp.{mi,psf})
    calexp[ccd].mi, sources[ccd] =
    ReinsertStars
    (calexp[ccd].{mi,psf}, sources[ccd])
    sources[ccd]
    DeblendSources
    = (sources[ccd], calexp.{mi,psf})
    sources[ccd]
    MeasureSources
    = (sources[ccd], calexp.{mi,psf})
    calexp.psf.apcorr
    FitApCorr
    (matches
    =
    , sources)
    parallel
    for
    ccd
    in
    SCIENCE_SENSORS :
    sources[ccd]
    ApplyApCorr
    =(sources[ccd], calexp.psf)
    return
    calexp , sources
    This is essentially just another
    BootstrapImChar
    iteration
    , without
    of
    the
    loop
    WCS-
    in
    fitting or artifact-handling stages. Previously-extracted
    used in PSF modeling, but we do not expect to do any additional
    sensors in this pipeline.
    Note that RefineImChar does not update the CalExp’s WCS,
    WCS and PhotoCalib will have already
    StandardJointCal
    been better
    , and no
    constrained
    changes have been made to the pixels. The Image is only updated
    The contents of this document are subject to configuration control by the LSST
    68

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    ground, and the Mask is only updated to indicate new detections.
    5.1.4 FinalImChar
    FinalImChar is responsible for producing the final PSF models
    While similar
    RefineImChar
    to, it is run after at least
    BackgroundMatchAn-
    one iteration of
    dReject
    and possibly
    UpdateMasks
    pipelines, which provide it with the final
    and mask.
    The steps in FinalImChar are
    RefineImChar
    identical
    , withto
    just
    those
    a few
    inexceptions:
    •The background is not re-estimated and subtracted.
    •The suite of plugin
    Single
    run
    Frame
    by Measurement
    is expanded to included all
    rithms indicated in the 12
    first
    . Thisshouldprovideallmeasurements
    column of Figure
    in
    the
    DPDD
    Source table description.
    •We also classify sources
    Single
    by delegating
    Frame Classification
    ,
    tofill the final
    Source table’s
    extendedness
    field. It is possible this
    RefineImChar
    will also be run
    and
    BootstrapImChar
    for diagnostic purposes.
    5.1.5 FinalJointCal
    FinalJointCal
    almost
    identical
    is StandardJointCal
    to
    , and the details of the differences
    pend on the approach to absolute calibration and the as-built
    pipelines. Because it is responsible for the final photometric
    form some steps that could
    StandardJointCal
    be omitted
    because
    fromthey have no impact
    on the ImChar pipelines. This could include a role in determining
    calibration of the survey, especially if a Gaia is relied
    together.
    There is no need for FinalJointCal to produce a new or updated
    its own internal use), as subsequent steps do not need one,
    catalog used by Alert Production will be derived from the
    updated WCS and PhotoCalib for each CalExp, with the PhotoCalib
    absolute as well as relative calibration.
    The contents of this document are subject to configuration control by the LSST
    69

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    As discussed5.1.2
    in,section
    this pipeline may require a multi-tract
    5.2 Image Coaddition and Image Differencing
    The next group of pipelines in a Data Release Production
    image differencing, which we use to separate the static sky
    of both astrophysical quantities and observational quantities.
    iteration between pipelines that combine images and pipelines
    images from each exposure. At each differencing step, we
    that are unique to a single epoch (whether artifacts, background
    sources); we use these characterizations to ensure the
    features that are common to all epochs. Variable objects
    this context, as our models of their effective coadded PSFs
    is included in those models.
    The processing flow in this pipeline group again centers
    CalExp dataset, which are limited here to its Background
    component is also updated, but only to subtract the updated
    to the previous pipeline group
    5.1
    todescribed
    update other
    in Section
    CalExp components.
    As in the previous pipeline group, tracts are processed independently,
    overlap multiple tracts, multiple CalExps (one for each
    these visits. The data flow between9,pipelines
    with the numbered
    is shown in
    steps
    described further below:
    1.The first version of the CalExp dataset
    BootstrapImChar
    is produced
    ,
    by
    StandardJointCal
    , and
    RefineImChar
    pipelines, as described
    5.1
    in Section
    2.We generate an updated
    BackgroundMatchAndReject
    and Mask via the
    pipeline. This produces the final CalExp Background and
    Mask.
    3.If the CalExp Mask has beenFinalImChar
    finalized,
    and
    FinalJointCal
    we run
    pipelines.
    the
    These produce the final PSF, WCS, and PhotoCal. If the Mask
    execute at least one iteration of the next step before
    4.We run WarpTemplates
    the
    ,CoaddTemplates
    , and
    DiffIm
    pipelines to generate the
    ASource and DiffExp datasets. We may then be able to generate
    The contents of this document are subject to configuration control by the LSST
    70

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    than we can obtain
    BackgroundMatchAndReject
    from
    by comparing the DiffExp masks
    across visits
    UpdateMasks
    in the
    pipeline.
    5.After all CalExp components have
    WarpRemaining
    been finalized,
    and
    Coad-we run
    dRemaining
    to build additional coadd data products.
    The baseline ordering of these steps is thus {1,2,3,4,5},
    and we may ultimately require an ordering that repeats steps
    ordering and number of iteration will require testing with
    taken with a realistic cadence; it is possible the configuration
    releases as the survey increases in depth. Fortunately,
    significant new algorithm development.
    This pipeline group is responsible for producing the following
    CalExp
    See above.
    DiffExp
    A CCD-level
    Exposure
    that is the difference between the CalExp and
    in the coordinate system of the CalExp. It may have the
    ditional PSF matching is used) or its own PSF model (if
    lated
    13
    after matching).
    DIASource
    SourceCatalog
    containing sources detected and measured
    ages.
    ConstantPSFCoadd
    A coadd data Exposure
    product
    or subclass
    (
    thereof) with a
    predefined PSF.
    DeepCoadd
    A coadd data product built to emphasize depth at the
    ing.
    BestSeeingCoadd
    A coadd data product built to emphasize image quality
    pense of depth. Depending on the algorithm used, this
    ShortPeriodCoadd
    A coadd data product built from exposures in a
    such as a year, rather than the full survey. Aside from
    use the same filter as DeepCoadd.
    13
    Decorrelated images
    refer here to a technique for convolving images by the transpose
    differencing them, and then deconvolving the transpose
    DMTN-015
    of the effective
    for more information.
    The contents of this document are subject to configuration control by the LSST
    71

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    LikelihoodCoadd
    A coadd formed by correlating each image with its
    bining them, used for detection and possibly building
    ShortPeriodLikelihoodCoadd
    Short-period likelihood coadds will also
    TemplateCoadd
    A coadd data product used for difference imaging
    order to produce templates appropriate for the level
    these coadds may require a third dimension in addition
    sions (likely either wavelength or a quantity that is
    The nature of these coadd data products depends critically
    efficient algorithms for optimal coaddition, and whether
    ence imaging. These algorithms are mathematically well-defined
    see
    DMTN-015
    for more information. We will refer to the coadds
    rithms as “decorrelated coadds”; a variant with constant
    related coadd”) is also possible. This choice is also mixed
    correct for differential chromatic refraction in difference
    correction involve templates that are the result of inference
    coaddition. The alternative strategies for using decorrelated
    A
    We use decorrelated coadds for all final coadd products.
    Coadd will be standard decorrelated coadds with a spatially-varying
    FCoadd and TemplateCoadd will be constant-PSF partially-decorrelated
    BestSeeingCoadd data product will be dropped, as it will
    This will make coadds more expensive and complex to build,
    development for coaddition, but will improve coadd-based
    easier to warm-start multi-epoch measurements. Difference
    more visits may be usable as inputs to templates due to
    cut.
    B
    We use decorrelated coadds for all coadds but TemplateCoadd.
    proved, and the additional computational cost of coaddition
    that is not run iteratively. Difference imaging may be
    eligible for inclusion in templates may be reduced. In
    options for building templates:
    B1
    Templates will be built as PSF-matched coadds, or a product
    The contents of this document are subject to configuration control by the LSST
    72

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    B2
    Templates are the result of inference on resampled exposures
    C
    We do not use decorrelated coadds at all. DeepCoadd, BestSeeingCoadd,
    riodCoadd will be direct coadds, and ConstantPSFCoadd
    Coaddition will be simpler and faster, but downstream
    sophistication, coadd measurements may be lower quality,
    ments may be more difficult to optimize. Here we again have
    templates as
    B: option
    C1
    Templates will be built as PSF-matched coadds, or a product
    C2
    Templates are the result of inference on resampled exposures
    It is also possible to combine multiple scenarios across
    may not need special templates to handle DCR in most bands,
    approach in those bands. The final selection between these
    on LSST data or precursor data with similar DCR and seeing,
    algorithms and some approaches to DCR correction may be
    algorithm development does not go well.
    Further differences in the pipelines themselves due to the
    lated coadds will be described in the sections below.
    5.2.1 WarpAndPsfMatch
    This pipeline resamples and then PSF-matches CalExp images
    level image with a constant PSF. The resampling and PSF-matching
    plished separately by
    Image
    delegating
    Warping
    and
    PSFto
    Homogenization
    the
    algorithmic
    components, respectively. These operations can also be
    the matched-to PSF is first transformed to the CalExp coordinate
    sampling yields a constant PSF in the coadd coordinate system).
    be necessary (or at least easier to implement) for undersampled
    It is possible these operations will be performed simultaneously
    nent; this could potentially yield improved computational
    properly track uncertainty. These improvements are unlikely
    because these images and the coadds we build from them will
    The contents of this document are subject to configuration control by the LSST
    73

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    grounds and find artifacts, and these operations only require
    tainty. However, other coaddition pipelines may require
    capable of warping and PSF-matching simultaneously, and
    bly use it here as well. Simultaneously warping and PSF matching
    computational performance improvements.
    The only output of the WarpAndPsfMatch
    Exposure
    pipeline
    intermediate
    is the MatchedWarp
    data product. It contains
    Exposure
    all
    components,
    of the usual
    which must be propagated
    through the image operations as well. There is a separate
    visit} combination, and these can be produced by running
    on each such combination. However, individual CCD-level
    ple patches, so I/O use or data transfer may be improved by
    instances for a given visit together.
    5.2.2 BackgroundMatchAndReject
    This pipeline is responsible for generating our final estimates
    dating our artifact masks. It is one of the most algorithmically
    Release Production from the standpoint of large-scale data
    working prototype has not yet been demonstrated except for
    scan observing strategy makes the problem easier. The algorithm
    sky where the set of input images is constant, and we do n
    in extending this to an algorithm that works across image
    likely to be the parallelization and data flow necessary
    grounds over a full tract. Separate tracts are stil processed
    The steps involved in background matching are described
    performed on the MatchedWarp images; these are all in the
    have the same PSF, so they can be meaningfully added and
    processing.
    1.We define one of the visits that overlap
    reference
    an area
    . At
    image
    of the
    least in the naive local specification of the algorithm,
    continuous over the region of interest.
    Build Background
    This is done
    Reference
    by
    pipeline, which must artificially (but reversibly) enforce
    The contents of this document are subject to configuration control by the LSST
    74

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    that stitches together multiple visits to form a single-epoch-deep
    we develop an approach for dealing with discontinuity
    2.We subtract the reference image from every other visit
    any artifical features due to the construction of the reference
    3.We run
    Source Detection
    the per-visit difference images to find artifacts
    sources. We do not generate a traditional catalog of these
    be used to generate improved CalExp masks; they will likely
    Footprints
    .
    4.We estimate the background on the per-visit difference
    Matched Background
    algorithmic
    Estimationcomponent. This difference
    should be easier to be model than a direct image background,
    free of sources and astrophysical backgrounds. This
    communication between patches to ensure that the background
    sistent in patch overlap regions.
    5.We build a PSF-matched coadd by adding all of the visit
    and subtracting all of the difference image backgrounds;
    only the reference image background, which
    Coadd
    we then model
    Background Estimation
    algorithmic component. This background
    involve communication between patches to ensure consistency.
    will be performed
    Coaddition
    byalgorithmic
    the
    component,
    Warpedwhile
    Imagethe
    Comparison
    component is used to generate new CalExp masks
    pixel, multi-visit histograms of image and mask values
    rejection) to distinguish transients and artifacts from
    6.We combine the relevant difference backgrounds with the
    form them back to the CalExp coordinate systems to compute
    for each CalExp.
    We are assuming in the baseline plan that
    WarpAndPsfMatch
    we can use a matched-to
    large enough to match all visit images to it without deconvolution.
    adversely affects subsequent
    BackgroundMatchAndReject
    processing
    , wein
    may need to de-
    velop an iterative approach
    WarpAndPsfMatch
    in which
    only
    we to
    apply
    better-seeing
    first, using a smaller
    BackgroundMatchAndReject
    target PSF, run
    on these, and then re-match
    everything to a larger target PSF and repeat with a larger
    lem would suggest
    DiffIm
    that
    and
    UpdateMasks
    the pipelines would be even better
    The contents of this document are subject to configuration control by the LSST
    75

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    artifacts, so a more likely mitigation strategy would be
    to after at least one iteration of those pipelines,
    9at as described
    the beginning5.2
    of Section
    The outputs of BackgroundMatchAndReject are updated Background
    for the CalExp product. Because it is not built with the
    calibration, the PSF-matched coadd built here is discarded.
    5.2.3 WarpTemplates
    This pipeline is responsible for generating the resampled
    used to build template coadds for difference imaging. The
    and the nature of its outputs depends on whether we are using
    Aat the beginning
    5.2
    ), PSF-matched
    of
    B1or
    coadds
    C1), or inferring
    (
    B2or
    templates
    C2).
    (
    If we are using decorrelated
    A), the
    coadds
    output
    (option
    is equivalent to the
    data product produced
    WarpRemaining
    by the
    pipeline (aside from differences
    state of the input CalExps), and the algorithm to produce
    •We correlate the image with its own
    Convolution
    PSF by delegating
    soft-
    Kernelsto
    ware primitive.
    •We resample the image byImage
    delegating
    Warping
    software
    to theprimitive.
    Here we should strongly consider developing a single algorithmic
    operations. These operations must include full propogation
    If we are not using decorrelated
    B1orC1), the coadds
    output (
    is equivalent to
    Warp data product, and the algorithm
    WarpAndPsfMatch
    is the
    pipeline.
    same as the
    We
    cannot reuse existing MatchedWarps simply because we need
    If we are inferring
    B2orC2
    templates
    ), this pipeline
    (
    is only responsible
    ducing an output equivalent to the DirectWarp
    WarpRemaining
    data product
    pipeline. This work is
    Image
    delegated
    Warping
    software
    to the
    primitive.
    The contents of this document are subject to configuration control by the LSST
    76

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    5.2.4 CoaddTemplates
    This pipeline generates the TemplateCoadd dataset used
    ence imaging. This may not be
    a젉
    simple
    (and possibly
    coadd,
    and
    ); in
    at order
    least to
    in
    correct for differential chromatic refraction during difference
    wavelength or airmass dimension to the usual 2-d image, making
    The size of the third dimension will likely be small, however,
    consider TemplateCoadd to be a small suite of coadds, in
    different sum of or fit to the usual visit-level images (the
    Most of the work is
    DCR-Corrected
    done by the
    Template
    algorithmic
    Generation
    component,
    but its behavior depends on which of the coaddition scenarios
    beginning of
    5.2
    ):
    Section
    A,B1,C1
    One or more coadd-like images (corresponding to different
    etc.) are created byCoaddition
    delegating
    algorithmic
    to the component to sum
    TemplateWarp images with
    A
    different
    only:
    coadded
    weights.
    images are then
    tially decorrelated to constant
    Coadd
    PSF
    Decorrelation
    by delegating
    algorithmic
    to the
    component.
    B2,C2
    The template is inferred from the resample visit images
    is yet to be developed.
    5.2.5 DiffIm
    In the DiffIm pipeline, we subtract a warped TemplateCoadd
    DiffExp image, where we detect and characterize DIASources.
    Production’s
    Alert Detection
    pipeline but may not be identical for several
    variant must be optimized for low latency, and hence may
    perfectly acceptable in DRP. In addition, the input CalExps
    in DRP, which may make some steps taken in AP unimportant
    However, we expect that the algorithmic components utilized
    used by AP.
    The steps taken by DRP DiffIm are:
    The contents of this document are subject to configuration control by the LSST
    77

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    1.Retrieve the DiffIm template appropriate for the CalExps
    handling a full visit at aTemplate
    time), delegating
    Retrieval
    algorithmic
    to the
    compo-
    nent. This selects the appropriate region of sky, and
    dimensional template dataset to a 2-d image appropriate
    2.(optional) Correlate the CalExp with
    Convolution
    its own PSF,
    Kernel
    delegating
    software primitive. This is the “preconvolution” approach
    makes PSF matching easier by performing PSF-correlation
    eliminating the need for deconvolution. This approach
    but still needs development.
    3.Resample the template to the coordinate system of the
    Image Warping
    software primitive.
    4.Match the template’s PSF to the CalExp’s PSF and subtract
    Image Subtraction
    algorithmic component.
    5.Run
    Source Detection
    the difference image. We correlate the image
    usingConvolution
    the
    software
    Kernelsprimitive unless this was done
    tion.
    6.(optional) Decorrelate the CalExp
    Difference
    by delegating
    Image Decorrelation
    to the
    algorithmic component.
    7.Run
    DiffIm Measurement
    on the difference image to characterize difference
    preconvolution is used but decorrelation is not, the
    sured using algorithms applied to standard images; alternate
    oped for some measurements, but perhaps not all.
    DiffIm can probably be run entirely independently on each
    tainly be taken in Alert Production. However, joint processing
    computationally efficient for at least some parts of template
    produce better results if a more sophisticated full-visit
    5.2.6 UpdateMasks
    UpdateMasks is an optional pipeline that is only run if DiffExp
    date CalExp masks. As such, it is not
    DiffIm
    run
    , and
    after
    is never
    the last
    runiteration
    if
    BackgroundMatchAndReject
    constructs the final CalExp masks.
    The contents of this document are subject to configuration control by the LSST
    78

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Like
    BackgroundMatchAndReject
    , UpdateMasks compares the histogram of
    particular spatial point to determine which masks correspond
    cal sources and artifacts; we want to reject both from coadds)
    able objects. This work
    Coaddition
    is.delegated to
    5.2.7 WarpRemaining
    This pipeline is responsible for the full suite
    Coad-
    of resampled
    dRemaining
    , after all CalExp components have been finalized. It
    of the following data products, depending on the scenario(s)
    Section
    5.2
    :
    LikelihoodWarp
    CalExp images are correlated with their own PSF,
    Convolution
    software
    Kernelsprimitive
    Imageand
    Warping
    software
    the
    primitive.
    lihoodWarp is computed in allCit
    scenarios,
    may not need
    butto
    inpropagate
    option
    uncertainty beyond the variance, as the resulting coadd
    MatchedWarp
    As in
    WarpAndPsfMatch
    , CalExp images are resampled then
    common PSF,
    Image
    using
    Warping
    and
    PSF Homogenization
    . MatchWarped is only pro-
    duced in option
    C.
    DirectWarp
    CalExp images are simply resampled, with no further
    ing
    Image Warping
    . DirectWarp is only produced
    C.
    in option
    Given that all of these steps involve resampling the image,
    tional reasons to do the resampling once up front, and then
    While this is mathematically possible for all of these cases,
    the PSF correlation step required for building LikelihoodWarps.
    5.2.8 CoaddRemaining
    In CoaddRemaining, we build the suite of coadds used for deep
    ject characterization. This includes the Likelihood, ShortPeriodLikelihood,
    ShortPeriod, and ConstantPSF Coadds.
    The algorithms again depend on the scenarios
    5.2
    outlined
    :
    at
    The contents of this document are subject to configuration control by the LSST
    79

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    A,B
    All non-template coadds are built from LikelihoodWarps.
    riodLikelihoodCoadds by simple coaddition
    Image
    of the LikelihoodWarps,
    Coaddition
    algorithmic component. We decorrelate
    Coadd Decorrela-
    these using
    tion
    algorithmic component to produce ShortPeriodCoadds,
    LikelihoodCoadds to produce the full LikelihoodCoadd.
    decorrelated to produce DeepCoadd and ConstantPSFCoadd.
    C
    WegenerateLikelihoodCoaddandShortPeriodLikelihoodCoadds
    as above (though the accuracy requirements for uncertainty
    ShortPeriodCoadd,DeepCoadd,andBestSeeingCoaddare
    nations of DirectWarp images,
    Image Coaddition
    again
    algorithmic
    using the component.
    ConstantPSFCoadds are built by combining MatchedWarps.
    These coadds must propagate uncertainty, PSF models (including
    photometric calibration (including spatial- and wavelength-dependent
    tion), in addition to pixel values.
    5.3 Coadd Processing
    In comparison to the previous two pipeline groups, the large-scale
    processing is relatively simple. All pipelines operate
    large-scale iteration between pipelines. These pipelines
    parallelization at a lower level, as they will frequently
    be expected to fit on a single core.
    Coadd processing begins
    DeepDetect
    with
    pipeline,
    the
    which simply finds above-threshold
    regions and peaks in multiple detection coadds.
    Deep-
    These are
    Associate
    , then deblended at the
    DeepDeblend
    pixel
    . The
    level
    deblended
    in
    pixels are
    sured
    MeasureCoadds
    in
    , which may also fit multiple objects simultaneously
    undeblended pixels.
    5.3.1 DeepDetect
    This pipeline simply
    Source
    runs
    Detection
    algorithmic
    the
    component on combinations
    LikelihoodCoadds and ShortPeriodLikelihoodCoadds, then
    liminary characterization on related coadds. These combinations
    The contents of this document are subject to configuration control by the LSST
    80

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    objects with different SEDs, and there are a few different
    we’ll produce (which are not mutually exclusive):
    •We could simply detect on each per-band LikelihoodCoadds
    •We could build a small suite of cross-band LikelihoodCoadds
    and artificial but approximately spanning SEDs (flat spectra,
    •We could build
    coadd
    single
    from the per-band coadds, which is
    objects the color of the sky noise, but may be close enough
    range of SEDs.
    Any of these combinations may also be used to combine ShortPeriodLikelihoodCoadds.
    We may also convolve the images further or bin them to improve
    extended objects.
    Actual detection on these images may be done with a lower
    threshold
    , to
    ofaccount
    5
    for loss of efficiency due using the incorrect
    filter.
    The details of the suite of detection images and morphological
    further algorithmic research on precursor data (or LSST/ComCam
    with at least approximately the right filter set.
    After detection, CoaddSources may be deblended
    Single
    and characterized
    Frame Deblending
    ,Single Frame Measurement
    , and
    Single Frame Classification
    algorithmic
    componentsonDeepCoaddandShortPeriodCoaddcombinations
    lihoodCoadd combinations used for detection. These characterizations
    CoaddSource tables) will DeepAssociate
    be discarded
    pipeline
    after
    isthe
    run, but may
    necessary to inform higher-level association algorithms
    acterization processing in this pipeline
    DeepAssociate
    will
    pipeline,
    be set by the
    but we do not expect it to involve significant new code beyond
    ImChar pipelines.
    The only output of DeepDetect is the suite of CoaddSource
    The contents of this document are subject to configuration control by the LSST
    81

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    image) containing
    Footprints
    (including their Peaks and any characterizations
    association).
    5.3.2 DeepAssociate
    In DeepAssociate, we perform a sophisticated spatial match
    Sources, generating tables of DIAObjects, Object candidates,
    Sources that will be used toMOPS
    construct
    .
    SSObjects in
    We do
    not
    include the Source table in this merge, as virtually
    trophysical objects better detected elsewhere. Non-moving
    objects (even variable non-transient objects) will be detected
    DeepDetect
    (as CoaddSources). Transients and fast-moving objects
    significance with significantly less blending
    DiffIm
    (as
    (and
    DIA-
    much easier
    Sources). While a small number of transient/moving Sources
    be detected in difference images due to extra noise from the
    impossible to recover without a large false positive rate
    table.
    The baseline plan for association is to first associate DIASources
    same approach used in AlertDIAObject
    Production
    Generation
    algorithmic
    (i.e. the
    com-
    ponent), then associate DIAObjects with the Object
    multiple CoaddSource
    Generation
    algorithmic component). DIASources not associated
    sidered candidates for merging SSObjects,
    MovingObjectPipeline
    which will happen
    pipeline.
    These association steps must be considerably more sophisticated
    ing; they must utilize the limited flux and classification
    to decide whether to merge sources detected in different
    physical models to be included in the matching algorithms
    •We must be able to associate the multiple detections that
    motion stars into a single Object.
    •We must not associate supernovae with their host galaxies,
    positions may be essentially the same.
    The contents of this document are subject to configuration control by the LSST
    82

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    To meet these goals (as well as similar ones which still need
    will have to
    multiple
    generate
    hypotheses for some blend families. Some
    hypotheses will beDeepDeblend
    rejected
    , while
    by
    others may be present in
    Object catalog (flags will be used to indicate different interpretations
    terpretation). This is a generalization of the simple parent/child
    different blend hypotheses in the2.3
    SDSS
    ). database (see Section
    It is possible that associations could be improved by doing
    (under the hypothesis that CoaddSource presence or absence
    ASource association). This is considered a fallback option
    dure described above cannot be made to work adequately.
    The output of the DeepAssociate pipeline is the first version
    superset of all Objects that will be characterized in later
    5.3.3 DeepDeblend
    This pipeline simply
    Multi-Coadd
    delegates to
    Deblending
    algorithmic
    the
    component
    blend all Objects in a particular patch, utilizing all non-likelihood
    yields
    HeavyFootprints
    containing consistent deblended pixels for
    likelihood) coadd, while rejecting as many deblend hypotheses
    ber of hypotheses that must be subsequently measured.
    While the pipeline-level code and data flow is simple, the
    only must deblending deal with arbirarily complex superpositions
    morphologies, it must do so consistently across bands and
    and ensure proper handling of Objects spawned by DIASources
    coadds. It must also parallelize this work efficiently over
    level images for all coadds in memory, the processing of
    families must themselves be parallelized. This may be done
    families into smaller groups that can be processed in parallel
    serial iteration; it may also be done by using low-level
    The output of the DeepDeblend pipeline is an update to the
    to indicate the origins of Objects and the decisions taken
    the set of rows to reflect the current object definitions. It
    The contents of this document are subject to configuration control by the LSST
    83

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    blend information to each Object. HeavyFootprints
    If stored,directly
    this would
    in the
    be a large dataset (comparable to the coadd pixel data).
    to the
    MeasureCoadds
    pipeline, but it almost certainly needs to be
    as well. Depending on the deblender implementation, it
    analytic models or some other compressed
    HeavyFootprints
    form that would
    to be reconstructed quickly on the fly, while requiring a
    per-object information. If this compression is lossy, it
    deblend results are
    MeasureCoadds
    first used
    so the
    in deblends used there can
    reconstructed later.
    5.3.4 MeasureCoadds
    The MeasureCoadds pipeline
    Multi-Coadd
    delegates
    Measurement
    to
    algorithmic
    the
    compo-
    nent to jointly measure all Objects on all coadds in a patch.
    Like
    DeepDeblend
    , this pipeline is itself quite simple, but it delegates
    component (but a simpler
    Multi-Coadd
    one than
    Deblending
    ). There are three classes
    questions in how multi-coadd measurement will proceed:
    •What parameters will be fit jointly across bands, and which
    measurement framework for multi-coadd measurement is
    fitting, but it is likely that some
    Single
    algorithms
    Frame Measurement
    will simply
    Forced Measurement
    plugins that are simply run independently
    and/or ConstantPSFCoadd in each band. Making these decisions
    tation on deep precursor and simulated data.
    •How will we measure blended objects? Coadd measurement
    the
    HeavyFootprints
    produced
    DeepDeblend
    by
    to useNeighborNoiseReplacement
    the
    approach, but we may
    Simultaneous
    then useto generate
    Fitting improved warm-start
    parameters
    MultiFit
    forto build models we can use as PSF-deconvolved
    enableDeblend
    the
    Template
    approach
    Projection
    MultiFit
    inand/or
    ForcedPhotome-
    try
    . If the deblender utilizes simultaneous fitting internally,
    the results of those fits directly as measurement outputs
    subsequent fitting that must be done.
    •How will we parallelize?
    DeepDeblend
    ,As
    keeping
    with the full suite of coadds
    ory will require processing at least some blend families
    The contents of this document are subject to configuration control by the LSST
    84

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    that don’t require joint fitting across different coadds,
    each coadd independently, but the most expensive algorithms
    are likely to be the ones where we’ll want to fit jointly
    The output of the MeasureCoadds pipeline is an update to the
    containing measured quantities.
    5.4 Overlap Resolution
    The two overlap resolution pipelines are together responsible
    Objects by merging redundant processing done in tract and
    cases, object definitions in the overlap region will be the
    even when the definitions are different we can frequently
    geometrical arguments. However, some difficult cases will
    families that are defined differently on either side.
    We currently assume that overlap resolution actually drops
    this will avoid redundant processing
    MultiFit
    pipeline.
    the performance
    A slower
    perhaps safer alternative would be to simply flag redundant
    tract overlap resolution
    MultiFit
    toand
    beForcedPhotometry
    moved after
    pipelines,
    the
    which
    would simplify large-scale parallelization and data flow
    more than one
    ResolveTractOverlaps
    tract (
    ) until after all image processing
    5.4.1 ResolvePatchOverlaps
    In patch overlap resolution, all contributing patches to
    four; see10
    Figure
    ) share the same pixel grid, and we furthermore
    the same coadd pixel values. This should ensure that any above-threshold
    is also above threshold in all others, which in turn should
    the extent of each blend familyFootprint
    (as defined
    ).
    by the parent
    A common pixel grid also allows us to define the overlap areas
    we consider each patch to have an inner region (which directly
    neighboring patches) and an outer region (which extends into
    patches). If we consider the case of two overlapping patches,
    The contents of this document are subject to configuration control by the LSST
    85

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    can fall into five different categories:
    •If the family falls strictly within one patch’s inner
    the other patch’s version of the family is dropped).
    •If the family crosses the boundary between patch inner
    ...but is strictly within both patches’ outer regions,
    inner region includes more of the family’s footprint
    ...but is strictly within only one patch’s outer region,
    ...and is not strictly within either patch’s outer
    merged at an Object-by-Object level. The algorithm
    to be developed, but will be
    Blended
    implemented
    Overlap
    by
    algo-
    Resolution
    the
    rithmic component.
    Overlap regions with more than two patches contributing
    qualitatively no different.
    Figure
    Patch
    10:
    boundaries and overlaps regions
    3 patches.
    for a single
    Differ-tract
    ent colors represent different patches; dashed lines show outer
    lines show inner patch regions. Light gray regions are processed
    medium regions as part of two, and dark regions as part of four.
    If pixel values in patch overlap regions cannot be guaranteed
    resolution becomes significantly harder (but no harder than
    The contents of this document are subject to configuration control by the LSST
    86

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    adjacent patches may disagree on the above categories to
    Patch overlap resolution can be run independently on every
    a different set of patches contributing to it; in the limit
    three times as many overlap regions as patches (each patch
    by two patches, and four overlap regions each shared by four
    5.4.2 ResolveTractOverlaps
    Tract overlap resolution operates under the same principles
    the fact that different tracts have different coordinate systems
    ues makes the problem significantly more complex.
    While we do not attempt to define inner and outer regions for
    crete overlap regions in which the set of contributing tracts
    must now be defined using spherical geometry). Because tracts
    membership of blend families, it will be useful here to define
    within an overlap region a family’s blend chain is the recursive
    with in any tract that contributes to
    11
    .that
    A blend
    overlap
    chain
    region
    is thus
    the maximal cross-tract definition of the extent of a blend
    categorize blends in tract overlaps:
    1.If a blend chain is strictly contained by only one tract,
    assigned to that tract. Note that this can occur even if
    tracts, as11
    in
    ; region
    Figure1 there is wholly contained only
    though it overlaps the green tract.
    2.If a blend chain is strictly contained by more than one
    are assigned to the tract whose center is closest to the
    is illustrated by 11
    region
    , which
    2 in
    would
    Figure
    be assigned to the red
    3.If a blend chain is not strictly contained by any tract,
    merged at an Object-by-Object Blended
    level. This
    Overlap
    is done
    Resolution
    by
    algorithmic component, after first transforming all measurements
    system defined to minimize distortion due to projection
    the blend chain’s centroid).
    The contents of this document are subject to configuration control by the LSST
    87

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    ResolveTractOverlaps is the first pipeline in Data Release
    cessed results from more than one tract.
    3
    2
    0
    1
    Figure
    Tract
    11:
    overlap scenarios, corresponding to the enumerated
    region outlined in black is a blend chain; transparent filled
    contributes from individual
    0is
    tracts.
    strictly
    The region
    contained
    labeled
    by the
    tract and does not touch any others, so it does not participate
    all.
    5.5 Multi-Epoch Object Characterization
    The highest quality measurements for the vast majority
    by the
    MultiFit
    and
    ForcedPhotometry
    pipelines. These measurements include
    motions and parallax, galaxy shapes and fluxes, and light
    sede many (but not all) measurements previously made on coadds
    using deep, multi-epoch information to constrain models
    CalExp (or DiffExp) images.
    The difference between the two pipelines is their
    Multi-
    parallelization
    Fit
    pipeline processes a single Object family at a time, utilizing
    that family asForcedPhotometry
    input, while
    processes one CalExp or DiffExp at
    ating over all Object families within its bounding box.
    perform three roles:
    The contents of this document are subject to configuration control by the LSST
    88

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    •Fit moving point source and galaxy models to all Objects,
    ing existing columns in the Object table. This requires
    ously, so it must
    MultiFit
    be.done in
    •Fit fixed-position point source models
    MultiFit
    -derived
    for each object
    posi-
    tions) to each DiffExp image separately, populating
    differ-
    the
    ential forced
    could
    photometry
    concievably
    MultiFit
    be
    , but
    done
    will
    in probably be
    efficient ForcedPhotometry
    do in
    .
    •Fit fixed-position point source models for each object
    also populating the ForcedSource
    direct forced
    table.
    can
    photometry
    easily
    This be done
    in either pipeline,
    MultiFit
    should
    but doing
    give
    itus more options for dealing
    ing, and it may decrease I/O costs as well.
    5.5.1 MultiFit
    MultiFit is the single most computationally demanding pipeline
    and its data flow is essentially orthogonal to that of all
    ing flow based on data products, each MultiFit job is an Object
    images, and hence efficient I/O will require the orchestration
    order that minimizes the number of times each image is loaded.
    From the Science Pipelines side, MultiFit is implemented
    orchestration layer:
    •The MultiFit “launcher” processes the Object table and
    including the region of sky required and the corresponding
    gions (unless the latter two are more efficiently derived
    tration layer).
    •The MultiFit “fitter” processes a single Object family,
    from the orchestration layer and returning an Object
    related ForcedSources).
    Multi-Epoch
    This is
    Measurement
    algorithmic
    the
    component.
    This simple picture is complicated by the presence of extremely
    Some blend families may be large enough that a single MultiFit
    The contents of this document are subject to configuration control by the LSST
    89

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    ory than is available on a full node (or require more cores
    lower-level parallelization). We see two possibilities
    •The fitter could utilize cross-node communication to extend
    most obvious approach would give each node full responsibility
    group of full CalExps it holds in memory, as well as responsibility
    ber of MultiFit jobs. These jobs would delegate pixel processing
    responsible for them (this constitutes the bulk of the
    low-latency but low-bandwidth communication; the summary
    tween the directing jobs and the CalExp-level processing
    actual CalExps or even the portion of a CalExp used by
    communication happens within a relatively tight loop
    This approach will also require structuring the algorithmic
    nication, and may require an alternate mode to run small
    •The launcher could define a graph of sub-family jobs that
    divide-and-conquer approach to large families. This
    ity in the algorithmic code to handle more combinations
    (to deal with neighboring objects on the edges of the
    tuning and experimentation, and more sophisticated launcher
    large objects in this scenario could also require binning
    data access layer.
    It is unclear which of these approaches will be more computationally
    option may reduce I/O or total network usage at the expense
    The second option may require redundant processing by forcing
    of iterative fitting may lead to faster convergence and hence
    If direct forced photometry is performed in MultiFit, moving-point
    be re-fit with per-epoch amplitudes allowed to vary independently
    held fixed. The same approach could be used to perform differential
    this would require also passing DiffExp pixel data to MultiFit.
    Significant uncertainty also remains in how MultiFit will
    but this decision will not have larger-scale processing
    in Section
    6.7.3
    The contents of this document are subject to configuration control by the LSST
    90

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    5.5.2 ForcedPhotometry
    In ForcedPhotometry, we simply measure point-source and
    (the baseline is point source photometry, but aperture photometry
    for diagnostic use and as a fallback) on individual CalExp
    from the Object table.
    Aside from querying the Object table for the list of Objects
    is delegated
    Forced
    to the
    Measurement
    algorithmic component. The only algorithmic
    lenge is how to deal with blending. If only differential forced
    pipeline, it may be appropriate to simply fit all Objects within
    point source models. The other alterative
    MultiFit
    or
    ispossibly
    to project
    MeasureCoadds
    and replace neighbors with noise
    6.7.3
    (as
    and
    6.7.3
    described
    ).
    5.6 Postprocessing
    The pipelines in the postprocessing group may be run after
    complete, and with the possible
    MakeSelectionMaps
    exception
    , include
    ofno image process-
    ing themselves. While we do not expect that these pipelines
    gorithm development, they include some of the least well-defined
    Production; many of these pipelines are essentially placeholders
    be split out into multiple new pipelines or included in existing
    a more detailed design here is blocked more by the lack of
    than a need for algorithmic research.
    5.6.1 MovingObjectPipeline
    The Moving Object Pipeline plays essentially the same role
    the SSObject (Solar System Object) table from DIASources
    ated with DIAObjects. We will attempt to make its implementation
    the
    AP Moving Object
    , but
    Pipeline
    the fact that DRP will run on all DIASources
    once (instead of incrementally) make this impossible in
    some iteration):
    •DelegateMake
    to the
    Tracklets
    algorithmic component to combine unassociated
    The contents of this document are subject to configuration control by the LSST
    91

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Sources
    tracklets
    into
    .
    •DelegateAttribution
    to the
    and
    algorithmic
    Precovery component to predict
    tions of known solar system objects and associate them
    of a “known” solar system object clearly depends on the
    external catalog or a snapshot of the Level 1 SSObject
    •Delegatetothe
    Orbit Fitting
    algorithmic component to merge unassociated
    tracks and fit orbits for SSObjects where possible.
    The choice of initial catalog largely depends on the false-object
    if the only improvements in data release production are
    SSObjects, using the Level 1 SSObject table could dramatically
    may also remove the possibility of removing nonexistent
    The DRP Moving Object Pipeline represents a full-survey
    but we expect that it will be a relatively easy one to implement,
    tively small inputs (unassociated DIASources) and produces
    its only major output (though IDs linking DIASources and
    either DIASource or a join table). This should mean that
    products have already been ingested, while requiring little
    the processing proceeds tract-by-tract.
    5.6.2 ApplyCalibrations
    The processing described in the previous sections produces
    be ingested into the public database: Source, DIASource,
    ForcedSource. The quantities inSource are either in raw
    tions in pixels) or pseudo-raw relative units (e.g. coadd-pixel
    These must be transformed into calibrated units via our
    tions, a process weRaw
    delegate
    Measurement
    to the algorithmic
    Calibration component.
    For the pseudo-raw relative units used for coadd measurements
    transformations are exact and hence do not introduce any
    applied.
    This is the primary place where the wavelength-dependent
    The contents of this document are subject to configuration control by the LSST
    92

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    ated by the Calibration Product Pipelines are applied.
    every object (or source) from its measured colors. The families
    color measurements used are subjects for future algorithmic
    ble to resolve these questions with relatively little effort.
    or deterministic, allowing science users to recalibrate
    One possible complication here is that PSF models are also
    SED for this purpose must be inferred much earlier in the
    desirable that the SEDs used for PSF-dependent measurement
    for photometric calibration, we may need to either infer
    preliminary color measurements or estimate the response
    PSF-evaluation SED so it can be approximately updated later.
    It is currently unclear when and where calibrations will
    •We could apply calibrations to tables before ingesting
    this would logically create new calibrated versions of
    •We could apply calibrations
    as
    we ingest
    tothem
    tables
    into the final database.
    •We could ingest tables into the temporary tables in the
    tions within the database.
    Regardless of which option is chosen
    Rawfor
    Measurement
    each public
    Calibra-
    table,
    tion
    algorithmic component will need to support operation
    memory table data and within the database (via, e.g. user-defined
    be needed to apply calibrations to intermediate data products
    the latter will be needed to allow Level 3 users to recalibrate
    assumed SEDs.
    5.6.3 MakeSelectionMaps
    The MakeSelectionMaps is responsible for producing multi-scale
    depth and efficiency at detecting different classes of object.
    be mapped, the format and scale of the maps (e.g. hierarchical
    the way the metrics will be computed are all unknown.
    The contents of this document are subject to configuration control by the LSST
    93

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    The approach must be extensible at Level 3: science users
    that can be utilized as efficiently by large collaborations
    the pressure on DM to provide a large suite of maps, but the
    still needs to be clarified to the community.
    One potential major risk here is that the most common way
    and selection metrics is to add fake sources to the data and
    reprocessing each unit of order 100 times. Because the reprocessing
    all processing steps (assuming the skipped steps can be
    not automatically be ruled out – if the pipelines
    DeepDetect
    ) that must
    are significantly faster than
    MultiFit
    skipped
    ), the overall
    steps (such
    impact
    as on processing
    could still be negligible. Regardless, the role of DM in
    to be clarified to the community.
    5.6.4 Classification
    In its simplest realization, this pipeline computes variability
    tic and/or discrete classification of each Object as a star
    include other categories (e.g. QSO, supernova).
    Variability summary statistics
    Variability
    are delegated
    Characterization
    algorithmic
    to the
    component.
    Type classification is
    Object
    delegated
    Classification
    algorithmic
    to the
    component. This
    utilize any combination of morphological, color, and variability/motion
    use spatial information such as galactic latitude as a Bayesian
    only morphology will also be available.
    Both variability and type classification may require “training”
    ject and ForcedSource tables and/or similar tables derived
    than imposing a full-survey sequence point here, we’ll probably
    results from a small-area validation release.
    The contents of this document are subject to configuration control by the LSST
    94

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    5.6.5 GatherContributed
    This pipeline is just a placeholder for any DM work associated
    validating major community-contributed data products.
    In addition to data products produced by DM, a data release
    products (essentially additional Object table columns)
    clude photometric redshifts and dust reddening maps. While
    to developing algorithms or code for these quantities, its
    tion and running user code at scale. The parties responsible
    and their relationship to DM needs to be better defined in terms
    including community-contributed data products in a data
    The contents of this document are subject to configuration control by the LSST
    95

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    6 Algorithmic Components
    This section describes mid-level Algorithmic Components
    contexts) by the pipelines
    3,4,described
    and
    5. These inSections
    turn depend on the
    lower-level Software Primitives
    7. Algorithmic
    described
    Components
    in Sectionwill
    be implemented as Python classes
    Task
    classes
    (such
    in the
    as the
    codebase as of
    this was written) that frequently delegate to C++. Unlike
    sections, which occupy a specific place in a production,
    designed to be reusable in slightly different contexts,
    them being used in one place. Many components may require
    different contexts, however, and these different variants
    classes. These context-specific variants are identified
    We expect that these components will form the bulk of the
    6.1 Reference Catalog Construction
    6.1.1 Alert Production Reference Catalogs
    Alert Production will use
    Object
    a table
    subsetas
    ofathe
    reference
    DRP
    catalog.
    Object
    table is regenerated on the same schedule as the template
    duction, we should always be able to guarantee that the reference
    images are consistent and cover the same area.
    Obtaining this catalog from the Level 2 database should simply
    query, though some experimentation and iteration may be
    6.1.2 Data Release Production Reference Catalogs
    The reference catalog used in Data Release Production is
    the Gaia catalog, but it may be augmented by data taken by LSST
    short-exposure, full-survey layer). DRP processing will
    lizing LSST survey data) in the course of a single production,
    these changes will be propagated to later data releases.
    The contents of this document are subject to configuration control by the LSST
    96

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    Constructing the DRP reference catalog is thus more of a one-off
    software component, and much of the work will be testing
    much we need to augment the Gaia catalog.
    6.2 Instrument Signature Removal
    Two variants of the instrument signature removal (ISR) pipeline
    with the difference arising from the real-time processing
    outlines the baseline design for DRP, with
    6.2.1
    the differences
    •Overscan subtraction: per-amplifier subtraction of the
    vector or array offset. After cropping out the first one
    potential contamination from CTI or transients, a clipped
    tracted if using a scalar subtraction (to avoid contamination
    trails), and a row-wise median subtracted if using a vector
    tion turns out to be necessary (unlikely, especially given
    frame later in the process), some thought should be given
    extra noise to the image.
    •Assembly: per-amplifier treatment of each CCD flavor (e2v
    differently) followed by per-CCD / per-raft assembly
    Application
    EDGE
    mask-bit
    of
    to appropriate
    i.e.
    regions
    around of the
    edges
    CCDs,
    of both sensor flavors, and around the midline region of
    from the anti-blooming implant.
    •Linearity: apply linearity
    master
    correction
    linearity
    , marking
    using
    table
    regions
    the
    where the linearity is considered
    SUSPECT
    .
    unreliable as
    •Gain correction: applied for CBP measurements where
    multiply
    absolute
    by the togains
    convert from ADUs to electrons, and
    pixel variance.
    •Crosstalk: Apply crosstalk correction to the raw data-stream
    appropriate version
    masterof
    crosstalk
    the.
    matrix
    •Mask defects and saturation:
    masterapplication
    defect
    and
    master
    list of
    saturation
    levels
    to set
    BAD/SAT
    thebit(s) in the maskplane.
    The contents of this document are subject to configuration control by the LSST
    97

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    •Full frame corrections:
    – Bias: Subtract
    masterthe
    bias
    . frame
    – Dark: Subtract an exposuremaster
    lengthdark
    multiple
    , perhaps
    frameof the
    including the slew time, depending on when the array is
    – Flats: Divide by the appropriate
    monochromaticmasterflats
    linear combination
    .
    – Fringe frames: this will involve the subtraction of
    combination
    monochromatic
    of
    to match
    flats
    night
    the sky’sand
    spectrum
    the filter in
    use at the time of observation, though the plan for how
    remains to be determined.
    •Pixel level corrections:
    – The “brighter-fatter effect”: Apply brighter fatter
    from
    4.3.15
    §
    Need to add proper section about this and reference
    trivial ISR algorithm. Just not sure where to put it.
    – Static pixel size effects: Correction of static effects
    etc.
    using data
    4.3.14
    from
    As§above - needs details and referencing.
    •CTE correction: The method used to correct for CTE will
    fully characterize the4.3.16
    charge
    ).
    transfer (see §
    •Interpolation over defects and saturation: interpolate
    using the PSF,
    INTERP
    andbit
    setin the mask plane.
    •Cosmic rays: identification
    6.3.1
    ),
    of interpolation
    cosmic rays (see
    over
    § cosmic
    using PSF, and
    CR/INTERP
    setting
    bit(s)
    of in the mask plane.
    •Generate snap difference: simple pixel-wise differencing
    and fast moving transients for removal is baselined,
    could be involved.
    5.1.1
    See also §
    •Snap combination: the baseline design is for a simple
    create the full-depth exposure for the visit. However,
    less simplistic treatment in the event that there is a
    the snaps arising from either the telescope
    e.g.
    pointing
    the
    dome/ground layer.
    The contents of this document are subject to configuration control by the LSST
    98

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    6.2.1 ISR for Alert Production
    The ISR for the AP pipeline differs slightly to the one used
    constraint.
    Crosstalk correction will be performed inside the DAQ, where
    will have been loaded. However, whilst there is therefore
    of network outages where the local data buffer overflows and
    lost, crosstalk correction would need to be applied.
    14
    Flat fielding will be performed as
    night
    for DRP,
    sky’sbut
    will
    spectrum
    because
    not be the
    available to AP, the fringe frame subtracted will either
    taken from an array of pre-computed composite fringe frames
    formed using PCA on-the-fly.
    6.3 Artifact Detection
    6.3.1 Cosmic Ray Identification
    The need for a morphological cosmic ray rejection algorithm
    Firstly, the science pipelines are explicitly required
    ditional way, i.e. one single continuous integration, and
    exposures are taken in series and then aggregatedOSS-REQ-0288
    in ISR
    luxury of having multiple exposures taken in quick succession
    not simply reject anything in the difference of the snaps
    able to utilize measurements on the snap differences to identify
    Given the need for the morphological cosmic ray detection
    adopt an algorithm similar
    photo
    that
    pipeline.
    used inThe
    thebaseline
    SDSS
    algorithm
    requires some modest knowledge of the PSF and looks for features
    the PSF. Qualitatively, this works well on a variety of data
    depending on the inputs.
    [33
    ] present an alternative algorithm that does not depend
    14
    This assumes that alerts are still being generated in this eventuality.
    The contents of this document are subject to configuration control by the LSST
    99

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    instead assumes that cosmic ray features will be sharp from
    different from the baseline is necessary, an
    33
    ]algorithm
    would be like
    an option for exploration.
    6.3.2 Optical ghosts
    We will have a set of optical ghost models. Some of these will
    (e.g. pupil ghost). Others will be a set of ghosts produced
    of source position and brightness. The structure of the
    using stacked, dithered star fields. The latter will likely
    measured using projectors.
    The stationary ghosts will need to be fit for since they will
    the pupil rather than on the brightness of a given source
    data necessary to compute the total flux over the focalplane
    production processing. Using
    먉the
    andfit
    the
    topredictions
    stationary models
    of the
    source ghosts,
    , we will construct a ghost image
    㸀萂
    Ⰰ栌
    where
    runs over the stationary
    runs
    ghost
    over
    models
    the sources
    and
    contributing
    single source ghosts. We can then correct the image by:
    㸀뀉缄뀉
    6.3.3 Linear feature detection and removal
    Satellite trails, out of focus airplanes, and meteors all
    cal images. The Hough
    13
    ] is
    Transform
    common tool
    [
    used in computer vision
    to detect linear features. Linear
    ,features
    the perpendicular
    are parameterized
    distance
    to the line from the
    , the
    origin
    the angle
    and
    with of
    the x-axis.
    ⤀툉ⴀ崌⨀
    space
    The
    is binned
    and each pixel in the image adds its flux to all the bins consistent
    bright linear features, the bin at the true location of the
    one bright pixel is contributing to that location in parameter
    polled, the highest bins correspond to the linear features
    The contents of this document are subject to configuration control by the LSST
    100

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    This works very well in high signal-to-noise images, but
    is also susceptible to bright point sources overwhelming
    The Hough transform is the correct tool for finding linear
    signal to noise. Since this is not always true in astronomical
    some form of a modified Hough transform that preferentially
    linear features in high dynamic range data.
    One could imagine a variety of ways
    6]to
    first
    do apply
    this. an
    For
    edge
    example,
    detec-
    tion algorithm to pull out linear features and then use the
    longer linear features. Another approach suggested by Steve
    nication) is to compute the PCA of pixel values in a localized
    features, there will be a high amount of correlation in a
    pixels. This effectively boosts the linear feature’s signal
    produce a linear feature mask by simply applying a threshold.
    The baseline for the science pipelines will be to use a modified
    and mask linear features in visit images.
    6.3.4 Snap Subtraction
    Cosmic Rays
    When subtracting snaps to form a visit, we will need
    morphological identifier like the one outlined above to identify
    there will be real transients and we still only want to pick
    will also help to have less crowding, so we should do CR rejection
    have it.
    Ghosts
    Snap differences will not help with ghosting as the ghosts
    perfectly.
    Linear features
    Snap differences will provide significant leverage
    tures. Since each segment will appear in at most one snap
    marked as detected in the difference images that are part of
    The contents of this document are subject to configuration control by the LSST
    101

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    regions. This technique will require running some sort of
    quirements will be less stringent since the image will be
    6.3.5 Warped Image Comparison
    Additional artifacts will be detected in DRP by comparing
    resampled to the same coordinate system.
    Snap
    This
    Subtraction
    is,similar
    but will operate quite differently in practice, in that we
    with the morphological detection stages; instead we assume
    morphologically will have already been detected.
    Instead, this stage will examine the full 3-d data cube (two
    epoch dimension) for outliers in the epoch dimension that
    sions. This is an extension of traditional coadd outlier-rejection,
    rejections of single pixels (or small groups of pixels) due
    obviously detect astrophysical transients as well as image
    able; this stage is responsible for determining which pixels
    of the static sky, and we want to reject astrophysical transients
    The largest challenge for this algorithm is probably handling
    sources that are the nevertheless present in most epochs.
    is more subjective, and we may need to modify our criteria
    an outlier.
    6.4 Artifact Interpolation
    This component is responsible for interpolating over small
    as cosmic rays. By utilizing the PSF model, this interpolation
    downstream algorithms do not need to worry about masked pixels
    not have a built-in formalismaperture
    for missing
    or
    fluxes
    second-moment
    data, such as
    shapes
    ). Interpolated pixels will also be masked (both as interpolated
    the reason why).
    This will likely use Gaussian processes, but an existing
    considered to be a placeholder, as it only interpolates
    The contents of this document are subject to configuration control by the LSST
    102

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    trails).
    Artifact interpolation will not handle regions significantly
    be either be subtracted or masked.
    6.5 Source Detection
    Detection is responsible for identifying new sources in
    positions significance estimates. We expect the same algorithm
    rithms) to be run on single-visit direct images, difference
    images, this must include detection of negative and dipole
    The output of detection
    Footprints
    (containing
    is a set of
    peaks).
    In the limit of faint, isolated objects and white noise, detection
    imum likelihood threshold for (at least) point sources,
    an image with its PSF and thresholding. Other approaches
    classes of objects, such as:
    •In crowded stellar fields, we expect to need to detect
    brightest objects at each iteration
    5.1.1
    ).
    (see e.g. Section
    •Optimal detection of diffuse galaxies may require correlating
    the PSF.
    •When blending or any significant sub-threshold objects
    ties may be sufficiently different from the usual assumptions
    detection to allow those methods, and an alternate approach
    •When processing preconvolved difference images or likelihood
    need to operate on images that have already been correlated
    •When operating on non-likelihood coadds and standard
    may need to operate on images with significant correlated
    In deep DRP processing
    5.3
    ), detection
    (Sectionis closely
    deep association
    tied
    and
    deep
    to the
    deblending
    algorithms, and may change significantly from the baseline
    The contents of this document are subject to configuration control by the LSST
    103

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    opments in those algorithms. For example, we may need to
    to these operations
    background
    (and estimation
    ) that essentially merges these
    algorithmic component with no well-defined boundaries.
    6.6 Deblending
    Deblending is both one of the most important and one of the
    lenges for LSST, and our plans for deblending algorithms
    project at this stage.
    The baseline interface takes
    Footprints
    as
    (including
    input a set
    peaks),
    of
    possibly
    from detections on several
    Object
    Generation
    images
    , and(see
    a set of images (related
    not necessarly identical to the set of deytection
    HeavyFootprints
    images).
    that contain the deblended images of objects. The tree may
    a sequence of blend hypotheses that subdivide an image into
    may be different
    HeavyFootprints
    for each deblended image (at least one
    making the size
    HeavyFootprints
    of all
    comparable to this size of the image
    coadds. Depending on the deblending algorithm chosen, a
    the deblend results may be possible (in
    HeavyFootprints
    that that
    to be
    it would
    regenerated quickly from the image data).
    As deblending may
    simultaneous
    involveof galaxy
    fittingand point source models,
    also output the parameters of these models directly as measurements,
    ating a pixel-level separation of neighboring objects that
    algorithms
    NeighborReplacement
    via
    .
    Deblending large objects is
    background
    also closely
    estimation
    . Some
    related
    science
    to cases
    (focusing on small, rare objects) may prefer aggressive
    astrophysical backgrounds such as intra-cluster light
    cases obviously care about preserving these structures (as
    which are frequently difficult to model parametrically).
    catalogs with different realizations of the background,
    smaller-scale astrophysical background features in the
    a way to express multiple interpretations of the sky.
    The baseline approach to deblending involves the following
    The contents of this document are subject to configuration control by the LSST
    104

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    1.Define a “template” for each potential object in the blend
    mately reproduces the image of the object).
    2.Simultaneously fit the amplitudes of all templates in the
    3.Remove redundant templates/objects according to some
    first step).
    4.Apportion each pixel’s flux to objects according to the
    scaled template at the position of that pixel divided
    templates.
    Regardless of the templates used, this approach strictly
    the morphology of even complex objects in the limit that
    complexity in this approach is of course in the definition
    dealing with redundancy.
    The deblender in the SDSS Photo pipeline uses a rotational
    plates directly from the images. This approach is probably
    the deeper, more blended regime of LSST, and hence we plan
    parametric models (both PSF-convolved and not). An ansatz
    a approximately uniform color over its image may also be
    investigate other less-parametric models such as Gaussian
    or splines. Hybrid approaches, such as using a symmetry ansatz
    blends and more constrained models for the rest, will also
    This approach yields exactly only one level of parent/child
    tree; each peak in a blend generates at most one child, and
    To extend this to the multi-level tree we expect to need to
    to repeat this approach at multiple scales – though it is
    treat each scale differently; some possibilities include
    spatial filters and building a tree of peaks based on their
    A key facet of any approach to deblending is to utilize the
    children that can be safely identified as unresolved. This
    that can operate in crowded stellar fields as effectively
    as the density of the field increases (either as detected
    The contents of this document are subject to configuration control by the LSST
    105

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management Science Pipelines
    LDM-151
    Design
    Latest Revision 2017-07-19
    position on the sky), we can increase the probability with
    solved. The simultaneous template fit then becomes a simultaneous
    if we iterate this procedure with detection (after subtracting