Data Products Definition Document

LargeSynopticSurveyTelescop

DataProductsDefinition

M.Jurić,T.Axelrod,A.C.Becker,J.Becla,

D.Ciardi,A.J.Connolly,G.P.Dubois-Felsmann,

M.Freemon,M.Gelman,M.Graham,Ž.Ivezić,

K.S.Krughoff,K-TLim,R.H.Lupton,F.Mueller,

M.Patterson,D.Petravick,D.Shaw,C.

J.Swinbank,J.A.Tyson,M.Wood-Vasey

LSE-163

LatestRevision:2017-07-01

ThisLSSTdocumenthasbeenapprovedasaContent-Controlled

subjecttoconfigurationcontrolandmaynotbechanged,altered,

withoutpriorapproval.Ifthisdocumentischangedorsuperseded,

retaintheHandledesignationshownabove.Thecontrolisonthe

withthisHandleintheLSSTdigitalarchiveandnotprintedversions.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

Abstract

Thisdocumentdescribesthedataproductsandprocessing

bytheLargeSynopticSurveyTelescope(LSST).

TheLSSTwilldeliverthreelevelsofdataproductsandLevel1(nightly)data

productswillincludeimages,differenceimages,catalogs

tectedindifferenceimages,andcatalogsofSolarSystem

purposeistoenablerapidfollow-upoftime-domainevents.Level2(annual)data

productswillincludewellcalibratedsingle-epochimages,

logsofobjects,sources,andforcedsources,enabling

domainscience.Level3(user-created)dataproductserviceswill

casesthatgreatlybenefitfromco-locationofuserprocessing

LSSTArchiveCenter.LSSTwillalsodevote10%ofobserving

specialcadence.Theirdataproductswillbecreated

hardwareasLevels1and2.Alldataproductswillbemade

friendlydatabasesandwebservices.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

ChangeRecord

VersionDateDescriptionOwnername

12013-10-07InitialreleaseMarioJuric

22016-09-26ImplementationofLCR-758UpdateData

ProductsDefinitionDocument,LSE-163

GregoryDubois-

Felsmann(LCR),Tim

Jenness(document),

RobertMcKercher

(Docushare)

3.02017-07-03ImplementationofLCR-962.ReferenceLSE-61

requirements.Minorcleanups.

TimJenness

Documentcurator:MarioJurić

Documentsourcelocation:

https://github.com/lsst/LSE-163

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

iii

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Contents

1 Introduction

1.1 The Large Synoptic...................

Survey Telescope

1.2 General Image Processing

............

Concepts for LSST

2 .....

1.3 Classes of LSST.....................

Data Products

2 General Considerations

2.1 Estimator and Naming

....................

Conventions

2.2 Image Characterization

......................

Data

2.3 Fluxes and.

Magnitudes

.......................

2.4 Uniqueness of IDs across

.................

database versions

2.5 Repeatability

.......................

of Queries

3 Level 1 Data Products

3.1 Overview

............................

3.2 Level 1 Data.......................

Processing

3.2.1 Difference Image

.....................

Analysis

3.2.2 Solar System Object

...................

Processing

3.3 Level 1.

Catalogs

.........................

3.3.1

DIASource

Table

........................

3.3.2

DIAObject

Table

........................

3.3.3

SSObject

Table

........................

3.3.4 Precovery Measurements

.....................

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

3.3.5 Reprocessing the

..................

Level 1 Data Set

3.4 Level 1 Image

.......................

Products

3.4.1 Visit

.........................

Images

3.4.2 Difference

.......................

Images

3.4.3 Image Differencing

...................

Templates

3.5 Alerts

DIASources

to.........................

3.5.1 Information Contained

..................

in Each Alert26

3.5.2 Receiving and Filtering

...................

the Alerts 27

4 Level 2 Data Products

4.1 Overview

............................

4.2 Level 2 Data.......................

Processing

4.2.1 Object Characterization

..................

Measures 32

4.2.2 Supporting Science Cases

........

Requiring 34

Full

.....

Posteriors

4.2.3 Source Characterization

.....................

4.2.4 Forced Photometry

.......................

4.2.5 Crowded Field

....................

Photometry

4.3 The Level 2

........................

Catalogs

4.3.1

Object

TheTable

.......................

4.3.2

Source

Table

.........................

4.3.3

ForcedSource

Table

.......................

4.4 Level 2 Image

.......................

Products

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

4.4.1 Visit

.........................

Images

4.4.2 Calibration

........................

Data

4.4.3 Coadded

.......................

Images

4.5 Data Release Availability

...........

and Retention Policies

.....

5 Level 3 Data Products and Capabilities

5.1 Level 3 Data Products and Associated

........

Storage

.....

Resources

5.2 Level 3 Processing

......................

Resources

5.3 Level 3 Programming Environment

..........

and Framework

.....

5.4 Migration of Level 3 data

.................

products to Level

522

6 Data Products for Special Programs

A Appendix: Conceptual Pipeline Design

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Data Products Definition Document

Preface

The purpose of this document is to describe the data products

Survey Telescope (LSST).

To a future LSST user, it should clarify what catalogs, image

can expect from LSST. To LSST builders, it provides direction

System Requirements Document to system design, sizing,

tain to the data products.

Though under strict changelivingdocument.LSSTwillundergoaperiod

control, this is a

constructionandcommissioninglastingnolessthanseven

surveyoperations.Toensuretheircontinuedscientific

LSSTDataProductswillbeperiodicallyreviewedandupdated.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

1Introduction

1.1TheLargeSynopticSurveyTelescope

LSSTwillbealarge,wide-fieldground-basedopticaltelescope

tipleimagescoveringtheskythatisvisiblefromCerroPachón

baselinedesign,withan8.4m(6.7meffective)primarymirror,

㌀fieldofview,anda

3.2Gigapixelcamera,willallowabout10,000squaredegrees

usingpairsof15-secondexposures,withtypical5栌depthforpointsourcesof툉꤄㌀㔀⼀㘀(AB).

Thesystemisdesignedtoyieldhighimagequalityaswell

metricaccuracy.Thetotalsurveyareawillinclude꤄30,000deg

㌀with夌㴀Ⰰ㐀㔀⼀㘀蔄,andwillbe

imagedmultipletimesinsixbands,픉젉툉줉��,coveringthewavelengthrange320–1050

amoredetailed,butstillconcise,summaryofLSST,please7]1.

Theprojectisscheduledtobegintheregularsurveyoperations

About90%oftheobservingtimewillbedevotedtoadeep-wide-fast

uniformlyobservea18,000deg

㌀regionabout1000times(summedoverall

theanticipated10yearsofoperations,andyieldacoadded툉꤄㌀㠀⼀㘀.Thesedatawill

resultincatalogsincludingover㐀㤀billionstarsandgalaxies,thatwill

theprimaryscienceprograms.Theremaining10%oftheobserving

specialprojectssuchasaVeryDeepandFasttimedomain

TheLSSTwillbeoperatedinfullyautomatedsurveymode.

CamerawillbeprocessedbyLSSTDataManagementsoftware

imagedastrophysicalsourcesandb)detectandcharacterize

observeduniverse.Theresultsofthatprocessingwillbe

objectsandthemeasurementsoftheirproperties,andprompt

inastrophysicalscenerydiscoveredbydifferencingincoming

imagesoftheskyinthesamedirection(templates,see§3.4.3).Measurementswillbeinternally

andabsolutelycalibrated.

Thebroad,high-level,requirementsforLSSTDataProductsaregivenLSSTScienceRe-

quirements(DocumentSRD;LPM-17).Thisdocumentlaysoutthespecificsofwhatthedata

productswillcompriseof,howthosedatawillbegenerated,

1http://ls.st/2m9

2Informallyknownas“DeepDrillingFields”.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

theflow-downfromtheLSSTSRDthroughtheLSSTSystemRequirementsDocument(theLSR;

LSE-29)andLSSTtheObservatorySystemSpecifications(OSS;LSE-30),totheLSSTDataMan-

agementSystemRequirements(DMSR;LSE-61),theUMLmodel(LDM-133),andthedatabase

schema(LDM-153).Throughoutthisdocumentmarginnotesareused

formalLSSTrequirementsandparametersassociatedwith

1.2GeneralImageProcessingConceptsforLSST

Arawimage(baselinedasapairofsuccessive15-second

bytheLSSTcamera,isprocessedbytheInstrumentSignature

duceasingle-visitimagewith,atleastconceptually,counts

ingthetelescopepupil(inreality,therearemanyadditional

fects,includingrandomcountingnoiseandvarioussubtle

duringsubsequentprocessing).Thissingle-visitimage

cessedVisitImage”anditsmaindatastructuresinclude

masks,alldefinedonperpixelbasis.AftertheISRstep

theirvariancearenotmodifiedanymore.Thesesingle-visit

producecoaddedanddifferenceimages.Therestoftheprocessing

basedinterpretationofimagingobservationsthatincludes

assumptions.

Thebasicinterpretationmodelassumesasumofdiscrete

andarelativelysmoothbackground.Thebackgroundhas

butionthandiscretesources,anditcandisplaybothspatial

Discretesourcescanvaryinbrightnessandposition.The

twosuccessiveobservations,lessormoremotionthanabout

urallyseparatesstarswithpropermotionsandtrigonometric

intheSolarSystem.Someobjectsthatvaryinbrightness

periodoftime(e.g.,supernovaeandothercosmicexplosions).

Theimageinterpretationmodelseparatestime-independent

porallychangingcomponent(“DC”and“AC”,notrespectively).oper-

ationallynorastrophysicallyassociatedwithdiscrete

coincident.

Images(aseriesofFootprints,whereFootprintisasetof

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

Figure1:OverviewofdataproductsproducedbyLSSTImagingProcessing

somethresholdlevelsetbynoiseproperties)ofdiscrete

els(§4.2.1).Atwo-componentgalaxymodelincludesalinearcombination

withtheirradialintensityvariationdescribedusingSersic

movingpointsourcemodelwithitsparallaxmotionsuperposed

Thismodelsharesmotionparametersacrossthesixbandpasses

ineachband,andthusincludes11freeparameters.Both

allobjects,exceptforfast-movingobjects(theSolarSystem

rately.Discreteobjectsdetectedindifferenceimageswillbemodeledusingthree

pointsourcemodel,atrailedpointsourcemodel,andapoint

1.3ClassesofLSSTDataProducts

ThemainLSSTdataproductsareillustratedinFigure1(seeAppendixforaconceptual

ofpipelineswhichwillproducethesedataproducts).LSST

two,somewhatoverlappinginscientificintent,typesof

1.Analysisofdifferenceimages,withthegoalofdetecting

calphenomenarevealedbytheirtime-dependentnature.

superimposedonbrightextendedgalaxiesisanexample

ingwillbedoneonnightlyordailybasisandresultinLevel1dataproducts.Level

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

productswillincludedifferenceimages,catalogsof

ages(DIASources),astrophysicalobjects

3theseareassociatedto(

DIAObjects),andSolar

Systemobjects(SSObjects4).ThecatalogswillbeenteredintotheLevel1databaseand

madeavailableinnearrealtime.Notifications(“alerts”)DIASourceswillbe

issuedusingcommunity-acceptedstandards

5within60secondsofobservation.

dataproductsarediscussedin§3.

2.Analysisofdirectimages,withthegoalofdetectingand

jects.Detectionoffaintgalaxiesondeepcoaddsand

isanexampleofthisanalysis.TheresultsareLevel2dataproducts.Theseproducts,

generatedandreleasedannually

6,willincludethesingle-epochimages,

catalogsofcharacterizedObjects(detectedondeepcoaddsaswellasindividual

7),

Sources8(detectionsandmeasurementsonindividualvisits),ForcedSources(con-

strainedmeasurementoffluxonindividualvisits).It

Level1dataproducts(see§3.3.5).Incontrasttothe“living”Level

updatedinreal-time,theLevel2databasesarestatic

Level2dataproductsarediscussedin§4.

Thetwotypesofanalyseshavedifferentrequirementsontimeliness.

sitionofobjectsmayneedtobeimmediatelyfollowedup,

lost.Thustheprimaryresultsofanalysisofdifferenceimages

DIASources–generallyneedtobebroadcastaseventalertswithin60secondsofendofOTT1

acquisition.Theanalysisofscience(direct)imagesisDMS-REQ-0004

apartofannualdatareleaseprocess.

Recognizingthediversityofastronomicalcommunityneeds,

cessingnotpartoftheautomaticallygeneratedLevel1and

3TheLSSThasadoptedthenomenclaturebywhichsingle-epochdetections

objectsarecalled

sources.Thereaderiscautionedthatthisnomenclatureisnotuniversal:detectionswhatLSST

callssources,andusethetermsourcesforwhatLSSTcallsobjects.

4SSObjectsusedtobecalled“MovingObjects”inpreviousversionsofthe

nameispotentiallyconfusingashigh-propermotionstarsaremoving

istheonebetweenobjectsinsideandoutsideoftheSolarSystem.

5Forexample,VOEvent,see

http://ls.st/4tt

6Exceptforthefirsttwodatareleases,whichwillbecreatedsixmonths

7TheLSSTtakestwoexposuresperpointing,nominally15secondsin

snaps.Forthe

purposeofdataprocessing,thatpairofexposureswilltypicallybe

calledavisit.

8Whenwritteninboldmonospacetype(i.e.,

\tt),ObjectsandSourcesrefertoobjectsandsourcesdetected

andmeasuredasapartofLevel2processing.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

10%ofitsdatamanagementsystemcapabilitiestoenabling

ofLevel3(user-created)dataproducts.Level3capabilities

greatlybenefitfromco-locationofuserprocessingand/or

ter.Thehigh-levelrequirementforLevel3isestablishedSRD.Theirdetails

arediscussedin§5ofthisdocument.

Finally,LSSTSurveySpecifications(§3.4ofLSSTSRD)prescribethat90%ofLSST

timebespentintheso-called“universalcadence”mode

vationswillresultinLevel1and2dataproductsdiscussed

observingtimewillbedevotedtospecialprograms,designedtoobtainimproved

ofinterestingregionsofobservationalparameterspace.툉꤄㌀㜀,

perexposure)observations,observationswithveryshort꤄1minute),andob-

servationsof“special”regionssuchastheEcliptic,Galactic

MagellanicClouds.Thedataproductsfortheseprograms

processingsoftwareandhardwareandpossessthegeneral

dataproducts,butmaybeperformedonasomewhatdifferent

cussedin§6.

2GeneralConsiderations

MostLSSTdataproductswillconsistofimagesand/orcatalogs.

andofferedtotheusersasrelationaldatabaseswhichtheywillbeabletoquery.

wasshowntoworkwellbypriorlargesurveys,forexample

Differentdataproductswillgenerallybestoredindifferent

dataproductswillbestoredinaLevel1database.Level2“universalcadence”

bestoredinaLevel2database.Theproductsforspecialprograms

manydifferentdatabases,dependingonthenatureoftheprogram.

Nevertheless,allthesedatabaseswillfollowcertainnaming

cusstheseinthesubsectionstofollow.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

2.1EstimatorandNamingConventions

Forallcatalogsdata,wewillemployaconventionwhere

thesuffixErr,whiletheestimatesofinherentwidthsofdistribution

havethesuffixSigma9.Thelatteraredefinedasthesquarerootsofthe

thequotedvalueofthequantityathand.DMS-REQ-0333

Unlessnotedotherwise,maximumlikelihoodvalues(called

quotedforallfittedparameters(measurements).Together

end-userapplywhateverpriortheydeemappropriatewhen

10.Where

appropriate,multipleindependentsamplesfromthelikelihood

terizedeparturesfromGaussianity.

Wewillprovidevaluesofloglikelihood,the氌㌀forthefittedparameters,andthe

datapointsusedinthefit.Databasefunctions(orprecomputedDMS-REQ-0331

frequentlyusedcombinationsofthesequantities(e.g.,氌㌀　씉켉윉).Thesecanbeusedtoassess

themodelfitquality.Notethat,iftheerrorsofmeasuredquantitiesare

likelihoodisrelatedtothe氌㌀as:

댉㸀

븎『

쬉

㈀

栌쬉蜄㌀攌뼎

exp

鈎

缄

氌㌀

㌀錎

(1)

wheretheindex쬉runsoveralldatapointsincludedinthefit.For氌㌀isdefined

as:

氌㌀㸀㈎

쬉輎

��쬉缄∂

栌쬉逎

㌀

ⴀ(2)

where∂��isthemeanvalueof��쬉.

Forfluxes,werecognizethatasubstantialfractionofastronomers

orsmarginalizedoverallotherparameters,trustingthe

prior11.Forexample,thisisnearlyalwaysthecasewhenconstructing

magnitudediagrams.WewillsupporttheseusecasesbyprovidingDMS-REQ-0331

columns,takingcaretonamethemappropriatelysoasto

9Given딉measurements,standarderrorsscaleas

딉缄㈀　㌀,whilewidthsremainconstant.

10There’satacitassumptionthataGaussianisareasonablygooddescription

theMLpeak.

11It’slikelythatmostcaseswillrequirejusttheexpectationvalue

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

age.Forexample,acolumnnamedgFluxmaybetheexpectationvalueofthe

whilegFluxMLmayrepresentthemaximumlikelihoodvalue.

2.2ImageCharacterizationData

RawimageswillbeprocessedtoremoveinstrumentalsignatureDMS-REQ-0069

erties,includingbackgrounds(bothduetonightskyandDMS-REQ-0327

DMS-REQ-0070

tionanditsvariation,photometriczero-pointmodel,and

DMS-REQ-0029

DMS-REQ-0030

ThatcharacterizationiscrucialforderivingLSSTcatalogs

willbekeptandmadeavailabletotheusers.Theexactformat

willdependonthefinaladoptedalgorithminconsultation

ensuretheformatsinwhichthesedataareservedaremaximallyDMS-REQ-0328

DMS-REQ-0072

Eachprocessedimage

12,includingthecoadds,willrecordinformation

“varianceplane”),aswellasper-pixelmasks(the“mask

determinethevalidityandusefullnessofeachpixelinestimating

thatareaofthesky.

Thisinformationwillbeper-pixel,andpotentiallyunwieldy

plantoinvestigateapproximateschemesforstoringmasks

toMangleorSTOMP),inadditiontostoringthemonaperpixelbasis.DMS-REQ-0326

2.3FluxesandMagnitudes

DMS-REQ-0043

Becausefluxmeasurementsondifferenceimages(Level1data3)areperformed

againstatemplateandthusrepresentafluxdifference,the

thedifferenceimagecanbenegative.Thefluxcanalsogo

presenceofnoise.Negativefluxescannotbestoredas(Pogson)logofaneg-

ativenumberisundefined.Wethereforeprefertostorefluxes,

databasetables

13.DMS-REQ-0347

Wequotefluxesinunitsof“maggie”.Amaggie,asintroduced

12Itisalsofrequentlyreferredtoas

calibratedexposure,fromtheButlerproducttypeofcalexp.

13Thisisagoodideaingeneral.E.g.,givenmulti-epochobservations,

ratherthanmagnitudes.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

flux.Itisdefinedsothatanobjecthavingafluxofonemaggie

hasanABmagnitudeofzero:

촉ꠉ꤉㸀缄㌀⼀㘀log㈀㄀⤀윉　maggie⨀(3)

Wechosetousemaggies(asopposedto,say,Jansky)toallow

twodistinctsourcesofphotometriccalibrationerror:the

ofthesurvey,andtheerrorinabsolutecalibrationthat

fluxofphotometricstandards.

Nevertheless,weacknowledgethatthelargemajorityof

nitudes.Forconvenience,weplantoprovidecolumnswith

14,where

valueswithnegativefluxwillevaluatetoNULL.Similarly,wewillprovidecolumns

pressedinJy(anditserrorestimatethatincludestherelative

contributions).

2.4UniquenessofIDsacrossdatabaseversions

DMS-REQ-0292

Toreducethelikelihoodforconfusion,allIDsshallbeunique

versions,otherthanthosecorrespondingtouniquelyidentifiable

sures).

Forexample,DR4andDR5(oranyother)releasewillshareObject,Source,DIA-

ObjectorDIASourceIDs(see§3and4forthedefinitionsofObjects,DIAObjects,etc.).

2.5RepeatabilityofQueries

DMS-REQ-0291

Werequirethatqueriesexecutedataknownpointintimeagain

berepeatableatalaterdate.Thispromotesthereproducibility

data.ItisofspecialimportanceforLevel1catalogs(§3)thatwillchangeonanightly

newtimedomaindataisbeingprocessedandaddedtothecatalogs.

Theexactimplementationofthisrequirementislefttothe

14Thesewillmostlikelybeimplementedas“virtual”or“computed”columns

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

team.Onepossibilitymaybetomakethekeytables(nearly)

havingtwotimestamps–createdTaianddeletedTai,sothatqueriesmaybelimitedWHERE

clause:

SELECT*FROMDIASourceWHERE’YYYY缄MM缄DD缄HH缄mm缄SS’

BETWEENcreatedTAIanddeletedTAI

or,moregenerally:

SELECT*FROMDIASourceWHERE”dataisvalidasofYYYY缄MM缄DD”

Aperhapslesserror-pronealternative,iftechnically

tualdatabasesthattheuserwouldaccessas:

CONNECTlsst缄dr缄5yyyy缄mm缄dd

SELECT*FROMDIASource

Thelattermethodwouldprobablybelimitedtonightlygranularity,

nismtocreatevirtualdatabases/viewson-demand.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

3Level1DataProducts

3.1Overview

Level1dataproductsarearesultofdifferenceimageanalysis3.2.1).Theyinclude

thesourcesdetectedindifferenceimages(DIASources),astrophysicalobjectsthat

associatedto(DIAObjects),identifiedSolarSystemobjects

15(SSObject),andrelated,broadly

defined,metadata(includingeg.,cut-outs

16).

DIASourcesaresourcesdetectedondifferenceimages(thosewith

먉　딉㼀퐉툉숉츉팉먉딉뤉aftercorrelationwiththePSFprofile,with퐉툉숉츉팉먉딉뤉definedintheSRDandtransSNR

presentlysetto5).Theyrepresentchangesinfluxwithrespect

withhighprobabilityofbeinginstrumentalnon-astrophysical

ically,aDIASourcemaybeanobservationofnewastrophysicalobject

thatpositioninthetemplateimage(forexample,anasteroid),

inanexistingsource(forexample,avariablestar).Their

presentinthetemplateimagereduceditsbrightness,or

complex(eg.,trailed,forasourcewithpropermotionapproaching꤄deg/day,or“dipole-like”,

ifanobject’sobservedpositionexhibitsanoffset–true

sitiononthetemplate).SomeDIASourceswillbecausedbybackgroundfluctuations;

퐉툉숉츉팉먉딉뤉㸀㘀,weexpectaboutonesuchfalsepositiveperCCD

typicalnight).Theexpectednumberoffalsepositives

verystrongfunctionofadopted퐉툉숉츉팉먉딉뤉:achangeof퐉툉숉츉팉먉딉뤉by0.5resultsinavariation

ofanorderofmagnitude,andachangeof퐉툉숉츉팉먉딉뤉byunitychangesthenumber

positivesbyabouttwoordersofmagnitude.

ClustersofDIASourcesdetectedonvisitstakenatdifferenttimesare

aDIAObjectoranSSObject,torepresenttheunderlyingastrophysicalDMS-REQ-0285

sociationcanbemadeintwodifferentways:byassumingthe

objectwithintheSolarSystemmovingonanorbitaround

17,orbyassumingitto

distantenoughtoonlyexhibitsmallparallacticandproper

18.Thelattertypeofassoci-

15TheSRDconsidersSolarSystemobjectorbitcatalogtobeaLevel2dataSRD,Sec3.5).Neverthe-

less,tosuccessfullydifferentiatebetweenapparitionsofknownSolarDIASources

weconsideritfunctionallyapartofLevel1.

16Small,㐀㄀뜀㐀㄀,sub-imagesatthepositionofadetectedsource.Alsoknown

postagestamps.

17Wedon’tplantofitformotionaroundotherSolarSystembodies;eg.,

lefttothecommunity.

18Where’small’issmallenoughtounambiguouslypositionallyassociate

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

ationisperformedduringdifferenceimageanalysisright

TheformerisdoneatdaytimebytheMovingObjectsProcessingMOPS),unlesstheDMS-REQ-0089

DIASourceisanapparitionofanalreadyknownSSObject.Inthatcase,itwillbeflagged

duringdifferenceimageanalysis.DMS-REQ-0002

Attheendofthedifferenceimageanalysisofeachvisit,we

forallnewlydetectedDIASources19.

3.2Level1DataProcessing

3.2.1DifferenceImageAnalysis

Thefollowingisahigh-leveldescriptionofstepswhichnightlydiffer-

enceimageanalysis(seeFigures3and5):

1.AvisitisacquiredandreducedtoasingleProcessedVisitImage(cosmicrayrejection,

instrumentalsignatureremoval

20,combiningofsnaps,etc.).DMS-REQ-0069

2.TheProcessedVisitImageisdifferencedagainsttheDIA-

Sourcesaredetected.Ifnecessary,deblendingwillbeperformed

parentblendandthedeblendedchildrenwillbemeasuredDIASources

(seenextitem),butonlythechildrenwillbematchedDIAObjectsandalertedon.

Deblendedobjectswillbeflaggedassuch.DMS-REQ-0010

3.Thefluxandshape

21oftheDIASourcearemeasuredonthedifference

tometryisperformedontheProcessedVisitImageattheDIASourceto

obtainameasureofthetotalflux.DMS-REQ-0269

4.TheLevel1database(see§3.3)issearchedforaDIAObjectoranSSObjectwithwhich

topositionallyassociatethenewlydiscoveredDIASource22.Ifnomatchisfound,a

DIAObjectiscreatedandtheobservedDIASourceisassociatedtoit.DMS-REQ-0273

DMS-REQ-0271

object.DMS-REQ-0285

19ForobservationsontheEclipticneartheoppositionSolarSystem

DIASourcecounts

and(untilthey’rerecognizedassuch)overwhelmtheexplosivetransient

toquicklyidentifythemajorityofSolarSystemobjectsearlyinthe

20Eg.,subtractionofbiasanddarkframes,flatfielding,badpixel/column

21The“shape”inthiscontextconsistsofweighted2ndmoments,aswellasfitstoatrailedsource

dipolemodel.

22Theassociationalgorithmwillguaranteethata

DIASourceisassociatedwithnotmorethanoneDIA-

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

5.IftheDIASourcehasbeenassociatedwithanSSObject(aknownSolarSystemobject)

itwillbeflaggedassuchandanalertwillbeissued.

daytime(seesection3.2.2).DMS-REQ-0274

6.Otherwise,theassociatedDIAObjectmeasurementswillbeupdatedwith

lectedduringprevious12months.Hence,thecomputedDIAObjects

havea“memory”ofpastdatathatdoesnotextendbeyond

23.Allaffected

columnswillberecomputed,includingpropermotions,DMS-REQ-0319

diaCharacterization-

Cutoff

7.TheLevel224databaseissearchedforthethreeneareststarsand

totheDIAObjectinObjectsouttosomemaximumradius.TheIDsofdiaNearbyObjMaxS-

tar

diaNearbyObjMax-

Galaxy

diaNearbyObjRadius

neighborObjectsarerecordedintheDIAObjectrecordandprovidedintheissued

alert(seebelow).

DMS-REQ-0271

8.Analertisissuedthatincludes:thenameoftheLevelDMS-REQ-0002

whenthisdatabasehasbeenqueriedtoissuethisalert,DIASourceID,theSSObject

IDorDIAObjectID25,nameoftheLevel2databaseandtheIDsofObjects,and

theassociatedsciencecontent(centroid,fluxes,low-order

etc.),includingallDIASourcesfromthelast12monthsthatarelinkedSSObjectdiaCharacterization-

Cutoff

orDIAObject.ThesciencecontentassociatedwiththeLevelObjectswillnot

beincluded.SeeSection3.5foramorecompleteenumeration.DMS-REQ-0274

9.ForDIAObjectsalloverlappingthefieldofview,includingthose

newDIASourcefromthisvisit,forcedphotometrywillbeperformed

(pointsourcephotometryonly).Thosemeasurements

flaggedDIASources26.NoalertswillbeissuedfortheseDIASources.DMS-REQ-0317

10.Within24hoursofdiscovery,precoveryPSFforcedphotometrywillbeperformedL1PublicT

anydifferenceimageoverlappingthepositionofnewDIAObjectstakenwithinthepast

30days,andaddedtothedatabase.AlertswillnotbeissuedprecoveryWindow

information.DMS-REQ-0287

ObjectorSSObject.Thealgorithmwilltakeintoaccounttheparallaxandproper

astheerrorsinestimatedpositionsofDIAObject,SSObject,andDIASource,tofindthemaximallylikely

MultipleDIASourcesinthesamevisitwillnotbematchedtothesameDIAObject.

23ThisrestrictionisremovedwhenLevel1processingisrerunduring

3.3.5

24Level2databaseisadatabaseresultingfromannualdatareleaseprocessing.

4fordetails.

25Weguaranteethatareceiverwillalwaysbeabletoregeneratethealert

includedtimestampsandmetadata(IDsanddatabasenames).

26Forthepurposesofthisdocument,we’retreatingthe

DIASourcesgeneratedbyforcedphotometryor

erymeasurementstobethesameasDIASourcesdetectedindifferenceimages(butflaggedappropriately).

logicalschema,thesemaybedividedintotwoseparatetables.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

Inadditiontotheprocessingdescribedabove,asmaller

ferenceimagesbelowthenominal퐉툉숉츉팉먉딉뤉㸀㘀thresholdwillbemeasuredandtransSNR

ordertoenablemonitoringofdifferenceimageanalysisquality.DMS-REQ-0270

Also,thesystemwillhavetheabilitytomeasureandalert

27numberofsources

detectedbelowthenominalthresholdforwhichadditional

a퐉툉숉츉팉먉딉뤉=3sourcedetectionnearagravitationalkeyhole

28maybehighlysignificant

assessingthedangerposedbyapotentiallyhazardousasteroid.

bedefinedbythestartofLSSToperations.

3.2.2SolarSystemObjectProcessing

ThefollowingwilloccurduringregularSolarSystemobject

29,aftera

nightofobserving;seeFigure6):DMS-REQ-0004

DMS-REQ-0089

1.TheorbitsandphysicalpropertiesofallSSObjectsre-observedontheprevious

recomputed.Externalorbitcatalogs(orobservations)

estimates.UpdateddataareenteredtotheSSObjectstable.DMS-REQ-0288

DMS-REQ-0273

2.AllDIASourcesdetectedonthepreviousnight,thathavenotbeenmatchedatahigh

confidenceleveltoaknownObject,DIAObject,SSObject,oranartifact,areanalyzed

potentialpairs,formingtracklets.

3.Thecollectionoftrackletscollectedoverthepast30

30issearchedforsubsets

ingtracksconsistentwithbeingonthesameKeplerianorbit

4.Forthosethatare,anorbitisfittedandanewSSObjecttableentrycreated.DIASource

recordsareupdatedtopointtothenewSSObjectrecord.DIAObjects“orphaned”bythis

unlinkingaredeleted.

31.

27Itwillbesizedfornolessthan

꤄10%ofaverageDIASourcepervisitrate.

28AgravitationalkeyholeisaregionofspacewhereEarth’sgravity

suchthattheasteroidwouldcollidewiththeEarthinthefuture.

29Notethatthere

isnostrictboundonwhendaytimeSolarSystemprocessing,justthat,averagedover

somereasonabletimescale(eg.,amonth),anight’sworthofobserving

inmovingobjectsmaytakelongertoprocess,whilenightswithless

requirementisonthroughput,notlatency.

30Theexacttimeperiodisadesignandcomputationalquestiondiscussed

LDM-156.

31SomeDIAObjectsmayonlybeleftwithforcedphotometrymeasurementsattheir

DIAObjects

areforce-photometeredonpreviousandsubsequentvisits);thesewill

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

5.PrecoverylinkingisattemptedforallSSObjectswhoseorbitswereupdatedin

cess.Wheresuccessful,SSObjects(orbits)arerecomputedasneeded.DMS-REQ-0286

3.3Level1Catalogs

Thedescribedalertprocessingdesignreliesonthe“living”

objectsandsourcesdetectedondifferenceimages.Atthe

32,thisdatabasewill

tablesofDIASources,DIAObjects,andSSObjects,populatedinthecourseofnightlyDMS-REQ-0269

DMS-REQ-0271

DMS-REQ-0273

differenceimageandSolarSystemobjectprocessing

33.Asthesegetupdatedandadded

theirupdatedcontentsbecomesvisible(query-able)immediately

34.DMS-REQ-0312

ThisdatabaseisonlylooselycoupledtotheLevel2database.Allofthecouplingisthrough

tionalmatchesbetweentheDIAObjectsentriesintheLevel1databaseandtheObjectsinthe

Level2databasedatabase.ThereisnodirectDIASource-to-Objectmatch:ingeneral,atim

domainobjectisnotnecessarilythesameastrophysical

thetwoarepositionallycoincident(eg.anasteroidoverlapping

datamodelemphasizesthathavingaDIASourcebepositionallycoincidentwithObjectdoes

notimplyitisphysicallyrelatedtoit.Absentotherinformation,theleast

modelrelationshipisoneofpositionalassociation,notphysicalidentity.

Thismayseemoddatfirst:forexample,inasimplecaseof

DIASourcestoObjectsisexactlywhatanastronomerwouldwant.That

failsinthefollowingscenarios:

•Asupernovainagalaxy.ThematchedobjectintheObjecttablewillbethegalaxy,

isadistinctastrophysicalobject.Wewanttokeepthe

nova(eg.,colors,thelightcurve)separatefromthose

•Anasteroidoccultingastar.Ifassociatedwiththestaronfirstapparition,

wouldneedtobedissolvedwhenthesourceisrecognized

asearlyasadaylater).

32Itwillalsocontainexposureandvisitmetadata,MOPS-specifictables,

troversial,implementation-dependent,orlessdirectlyrelevantfor

document.

33Thelatterisalsocolloquiallyknownas

DayMOPS.

34Nolaterthanthemomentofissuanceofanyeventalertthatmayrefer

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

•Asupernovaontopofapairofblendedgalaxies.Itisnotclearingeneraltowhich

thisDIASourcewould“belong”.Thatinitselfisaresearchquestion.

DIASource-to-Objectmatchescanstillbeemulatedviaatwo-steprelationDIASource-DIAObject-

Object).Foreaseofuse,viewsorpre-builttablewiththese

end-users.DMS-REQ-0324

Inthesectionstofollow,wepresenttheconceptualschemasforthemostimportantLevel

1databasetables.Theseconveywhatdatawillberecordedineachtable,

detailsofhow.Forexample,columnswhosetypeisanarray(eg.,radec)maybeexpanded

toonetablecolumnperelementofthearray(eg.,ra,decl)oncethisschemaistranslated

SQL35.Secondly,thetablestobepresentedarelargelynormalized

information).Forexample,sincethebandofobservationDIA-

Sourcetabletothetablewithexposuremetadata,there’snobandintheDIA-

Sourcetable.Intheas-builtdatabase,theviewspresented

denormalizedforeaseofuse.

3.3.1

DIASource

Table

This is a table of sources

퐉툉숉츉팉먉딉뤉 툄 㘀

detected

on difference

(

DIASources

images

). On

transSNR

LSR-REQ-0101

average,

the

SRD

expects

LSST

꤄

10,000

astrophysical

DIASources

per visit

꤄

10M per

(

night; 100,000 per deg

㌀

of the sky per hour).

transN

Some

퐉툉숉츉팉먉딉뤉 툄 㘀

sources will not be caused by observed astrophysical

by artifacts (bad columns, diffraction spikes, etc.). The

attempt to identify and flag these as such.

Unless noted otherwise,

DIASource

quantities

all

(fluxes, centroids, etc.)

difference image.

DMS-REQ-0269

The SQL realization of this schema is defined in the

cat

package and documented

LDM-153

andin

can be

browsed

http://ls.st/8g4

This requirement is

specified

in the LSST

SRD

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

DIASource

2: Table

Name

Type

Unit

Description

diaSourceId

uint64

Unique source identifier

ccdVisitIduint64

ID of CCD and visit where this

was measured

diaObjectId

uint64

ID ofDIAObject

the this source was as-

sociated with, if any.

ssObjectIduint64

ID ofSSObject

the this source has been

linked to, if any.

parentDiaSourceId uint64ID of the

DIASource

parent

this object

has been deblended from, if any.

midPointTai

double time

Time of mid-exposure for this

Source

radec

double[2]

degrees

Centroid,

⤀嘌ⴀ 夌⨀

radecCov

float[3]variousradec

covariance matrix.

float[2]pixels Column and row of the centroid.

xyCov

float[3]variousCentroid covariance matrix.

apFlux

float

nmgy

Calibrated aperture flux. Note

this actually measures

differ-

the flux

ence

between the template and

visit image.

apFluxErr float

nmgy

Estimated uncertainty

apFlux

SNR

float

The signal-to-noise ratio at

source was detected in the difference

image.

Continued on next page

The visit mid-exposure time generally depends on the position of

motion trajectory.

The astrometric reference frame will be chosen closer to start of operations.

This is not necessarily the same as apFlux/apFluxErr, as the flux measurement

than the detection algorithm.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

DIASource

2: Table

Name

Type

Unit

Description

psFlux

float

nmgy

Calibrated flux for point source

Note this actually measures

difference

between the template and

the visit image.

psRadec

double[2]

degrees

Centroid for point source model.

psCov

float[6]variousCovariance matrix for point

model parameters.

psLnL

float

Natural

찉켉젉

likelihood of the observed

data given the point source model.

psChi2

float

氌

㌀

statistic of the model fit.

psNdata

int

The number of data points (pixels)

used to fit the model.

trailFlux float

nmgy

Calibrated flux for a trailed

model

ⴀ

. Note this actually mea-

sures the

difference

fluxbetween the

template and the visit image.

trailRadecdouble[2]

degrees

Centroid for trailed source model.

trailLength

float

arcsec Maximum likelihood fit of

length

ⴀ

trailAnglefloat

degreesMaximum likelihood fit of the

between the meridian through

centroid and the trail direction

ing, direction of motion).

Continued on next page

A “maggie”, as introduced by SDSS, is a linear measure of flux in units

magnitude

촉

ꠉ꤉

㸀缄㌀⼀㘀

log

㈀㄀

⤀

maggie

⨀

. “nmgy” is short for a nanomaggie

愌넉��

). For

(1example,

nmgy = 3.631

flux

㄀⼀㈀㘀㤀of

nmgy corresponds to AB

㌀㔀⼀㘀

magnitude

See

2.3

for details.

Trailed Source Model

attempts to fit a (PSF-convolved) model of a point source

amount in some direction (taking into account the two-snap nature of

around the mid-point of the trail). Roughly, it’s a fit to a PSF-convolved

fast-moving Solar System objects.

This model does not

direction

fitof

for

motion;

the

to recover it, we would need to fit the

to individual snaps of a visit. This adds to system complexity, and

performance given the added information.

Note that we’ll likely measure trailRow and trailCol, and transform

for storage in the database. A stretch goal is to retain both.

TBD: Do we need a separate trailCentroid? It’s unlikely that we do,

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

DIASource

2: Table

Name

Type

Unit

Description

trailCov

float[15]

variousCovariance matrix of trailed

model parameters.

trailLnL

float

Natural

찉켉젉

likelihood of the observed

data given the trailed source

trailChi2 float

氌

㌀

statistic of the model fit.

trailNdataint

The number of data points (pixels)

used to fit the model.

dipMeanFlux

float

nmgy

Maximum likelihood value for

mean absolute flux of the two

for a dipole

. model

dipFluxDifffloat

nmgy

Maximum likelihood value for

ference of absolute fluxes of

lobes for a dipole model.

dipRadec

double[2]

degrees

Centroid for dipole model.

dipLength float

arcsec Maximum likelihood value for

separation in dipole model.

dipAngle

float

degreesMaximum likelihood fit of the

gle between the meridian through

the centroid and the dipole direction

(bearing, from negative to

lobe).

dipCov

float[21]

variousCovariance matrix of dipole model

rameters.

dipLnL

float

Natural

찉켉젉

likelihood of the observed

data given the dipole source

dipChi2

float

氌

㌀

statistic of the model fit.

dipNdata

int

The number of data points (pixels)

used to fit the model.

Continued on next page

Dipole

attempts

Model

to fit a (PSF-convolved) model of two point sources,

arated by a certain amount in some direction. The primary use case is to

with image differencing (e.g., due to astrometric offsets).

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

DIASource

2: Table

Name

Type

Unit

Description

totFlux

float

nmgy

Calibrated flux for point source

measured on the visit image

tered at the centroid measured

difference image (forced photometry

flux)

totFluxErrfloat

nmgy

Estimated uncertainty

fpFlux

diffFlux

float

nmgy

Calibrated flux for point source

centered

radec

but measured on

the difference of snaps comprising

this

visit

diffFluxErrfloat

nmgy

Estimated uncertainty

diffFlux

fpBkgd

float

nmgy/asec

㌀

Estimated background at the position

(centroid) of the object in the

image.

fpBkgdErr float

nmgy/asec

㌀

Estimated uncertainty of

fpBkgd

Ixx

float

asec

㌀

Adaptive second moment of

source intensity. See Bernstein

Jarvis

2] for

[ detailed discussion

all adaptive-moment related

ties

Iyy

float

asec

㌀

Adaptive second moment of

source intensity.

Ixy

float

asec

㌀

Adaptive second moment of

source intensity.

Icov

float[6]asec

㔀

Ixx

Iyy

Ixy

covariance matrix.

IxxPSF

float

asec

㌀

Adaptive second moment for the

IyyPSF

float

asec

㌀

Adaptive second moment for the

IxyPSF

float

asec

㌀

Adaptive second moment for the

Continued on next page

This flux can be used to detect sources changing on timescales comparable

꤄

15sec).

http://ls.st/5f4

for a brief summary.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

DIASource

2: Table

Name

Type

Unit

Description

extendedness

float

A measure of extendedness,

puted using a combination of

able moments, or from a likelihood

ratio of point/trailed source

(exact algorithm

옉��퐉옉츉씉옉씉츉옉팉팉 㸀

TBD).

㈀

implies a high degree of confi-

dence that the source is extended.

옉��퐉옉츉씉옉씉츉옉팉팉 㸀㄀

implies a high degree

of confidence that the source

like.

spuriousness

float

A measure of spuriousness,

puted using information

from the

source and image characterization,

well as the information on the

scope and Camera system (e.g.,

maps, defect maps, etc.).

flags

bit[64]bit

Various useful flags.

Some fast-moving, trailed, sources may be due to passages

may exhibit significant curvature. While we do not measure

inferred by examining the length of the trail, the trailed

adaptive shape measures. Once curvature is suspected, the

to the cutout provided with the alert.

3.3.2

DIAObject

Table

DMS-REQ-0271

DMS-REQ-0272

diaNearbyMaxObj

The computation of spuriousness will be “prior free” to the extent possible

the astrophysical neighborhood of the source, whether it has been previously

to avoid introducing a bias against unusual sources or sources discovered

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

DIAObject

3: Table

Name

Type

Unit

Description

diaObjectId

uint64

Unique identifier.

radec

double[2]

degrees

⤀嘌ⴀ 夌⨀

position of the object at

radecTai

radecCov

float[3]variousradec

covariance matrix.

radecTai

double time

Time at which the object was at

sition

radec

float[2]mas/yr Proper motion

. vector

parallax

float

mas

Trigonometric arallax.

pmParallaxCov

float[6]

variousProper motion - parallax covariances.

pmParallaxLnL

float

Natural log of the likelihood

ear proper motion-parallax fit

pmParallaxChi2

float

氌

㌀

statistic of the model fit.

pmParallaxNdata int

The number of data points used

the model.

psFluxMeanfloat[ugrizy] nmgy

Weighted mean of point-source

flux,

psFlux

psFluxMeanErr

float[ugrizy]

Standard

nmgy

error

psFluxMean

psFluxSigma

float[ugrizy] nmgy

Standard deviation of the distribution

psFlux

psFluxChi2float[ugrizy]

氌

㌀

statistic for the scatter

psFlux

around

psFluxMean

psFluxNdata

int[ugrizy]

The number of data points used

compute

psFluxChi2

fpFluxMeanfloat[ugrizy] nmgy

Weighted mean of forced photometry

flux,

fpFlux

fpFluxMeanErr

float[ugrizy]

Standard

nmgy

error

fpFluxMean

fpFluxSigma

float[ugrizy] nmgy

Standard deviation of the distribution

fpFlux

Continued on next page

High proper-motion or parallax objects will appear as “dipoles” in

be taken not to misidentify these as subtraction artifacts.

radec

, and

parallax

will all be simultaneously fitted for.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

DIAObject

3: Table

Name

Type

Unit

Description

lcPeriodicfloat[6

뜀

32]

Periodic features extracted

light-curves using generalized

Scargle periodogram

]

. [Table 4,

lcNonPeriodic

float[6

뜀

20]

Non-periodic features extracted

light-curves

].[Table 5,

nearbyObj uint64[6]

Closest

Objects

(3 stars and 3 galaxies)

in Level 2 database.

nearbyObjDist

float[6]

arcsec Distances

nearbyObj

to.

nearbyObjLnP

float[6]

Natural

log

of the probability that

observed

DIAObject

is the same as the

nearby

Object

flags

bit[64]bit

Various useful flags.

3.3.3

SSObject

Table

DMS-REQ-0273

Table

SSObject

4: Table

Name

Type

Unit

Description

ssObjectIduint64

Unique identifier.

double[7]

various

Osculating orbital elements

(

턉

옉

줉

訂

渌

됉

㄀

, epoch).

oeCov

double[28]

various

Covariance matrix

for

arc

float

days

Arc of observation.

orbFitLnL float

Natural log of the likelihood

bital elements fit.

orbFitChi2float

氌

㌀

statistic of the orbital elements

orbFitNdata

int

The number of data points (observa-

tions) used to fit the orbital

Continued on next page

The exact features in use when LSST begins operations are likely to

described here. This is to be expected given the rapid pace of research

number

ofcomputedfeaturesisunlikelytogrowbeyondthepresentestimate.

This quantity will be computed by marginalizing over the product of

of the

Object

and

DIAObject

, assuming an appropriate prior.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table

SSObject

4: Table

Name

Type

Unit

Description

MOID

float[2]AU

Minimum orbit intersection

tances

moidLon

double[2]

degrees

MOID longitudes.

float[6]mag

Mean absolute magnitude, per

[10

, magnitude-phase system].

float[6]mag

급

㈀

slope parameter,

,per band

magnitude-phase system].

float[6]mag

급

㌀

slope parameter,

,per band

magnitude-phase system].

hErr

float[6]mag

Uncertainty of H estimate.

g1Err

float[6]mag

Uncertainty

급

㈀

estimate.

g2Err

float[6]mag

Uncertainty

급

㌀

estimate.

flags

bit[64]bit

Various useful flags.

The

급

㈀

and

급

㌀

parameters for the large majority of asteroids will

later in the survey. We may decide not to fit for it at all

later in Operations, or provide

급

㈀㌀

fits. Alternatively,

two-parameter we may fit it

priors on slopes poorly constrained by the data. The design

is insensitive to this decision, making it possible to postpone

follows the standard community practice at that time.

The LSST database will provide functions to compute

嘌

the phase

for every observation, as

꼉⤀嘌⨀

well

, andas

absolute,

the

꼉

, reduced,

asteroid magnitudes

LSST bands.

DMS-REQ-0323

3.3.4 Precovery Measurements

When a new

DIASource

is detected, it’s useful to perform PSF photometry

the new source on images taken prior to discovery.

precovery

These are

measurements

. Performing precovery in real time over all previously

DMS-REQ-0287

DMS-REQ-0286

http://www2.lowell.edu/users/elgb/moid.html

When Solar System objects are concerned, precovery has a slightly different

of newly identified

SSObjects

on previously acquired visits, and

DIASources

associating

consistent

with

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

I/O intensive to be feasible. We therefore plan the following:

1.For all newly discovered objects, perform precovery PSF

the previous 30 days

precoveryWindow

2.Make available a “precovery service” to request

DIA-

precovery

Sources

across all previous visits, and make it available

Web interface and machine-accessible APIs will

DMS-REQ-0341

be provided.

The former should satisfy the most common use cases (eg.,

an opportunity for more extensive yet timely precovery of

3.3.5 Reprocessing the Level 1 Data Set

In what we’ve described so far, the “living” Level 1 database

new images are

DIASources

taken identified.

and

Every

DIASource

time

isaassociated

new DMS-REQ-0312

to an existing

DIAObject

, the

DIAObject

record is updated to incorporate new

brought in

DIASource

by the

. Once discovered and

DIASources

measured,

would the

never

be re-discovered and re-measured at the pixel level.

This would be far from optimal. The instrument will be better

versions of LSST pipelines will improve detection and measurements

precovery photometry should optimally

objects,

be and

performed

not just

for

few. This argues

reprocessing

for periodic

of the Level 1 data set.

DMS-REQ-0313

DMS-REQ-0325

We plan to reprocess all image differencing-derived data

time we perform the annual Level 2 data release productions.

taken since the start of survey operations, to the time when

gins. The images will be reprocessed using a single version

measurement software, resulting in a consistent data set.

There will be three main advantages of Level 1 database produced

cessing, compared to “living” Level 1 database: i) even the

with these predictions.

We will be maintaining a cache of

㐀㄀

days of processed images to support this feature.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

the latest software, ii) astrometric and photometric calibration

be no 12-month limit on the width of data window

DIAObject

used to computed

measurements (proper motions, centroids, light curves,

diaCharacterization-

Cutoff

Older versions of the Level 1 database produced during Data

following the same rules as for the Level 2 databases. The

mate data release will be kept on disk and loaded into

DMS-REQ-0077

the

to tape and made available as4.5

bulk

for more

downloads.

detail.

See §

DMS-REQ-0300

3.4 Level 1 Image Products

3.4.1 Visit Images

DMS-REQ-0069

Raw and Processed Visit Images will be made available

L1PublicT

for

from the end of visit acquisition.

The images will remain accessible with low-latency (seconds

load) for at least 30 days, with slower access afterwards

l1CacheLifetime

3.4.2 Difference Images

DMS-REQ-0010

Complete difference images will be made available for

L1PublicT

download

the end of visit acquisition.

The images will remain accessible with low-latency (seconds

load) for at least 30 days, with slower access afterwards

l1CacheLifetime

3.4.3 Image Differencing Templates

DMS-REQ-0280

Templates for difference image analysis will be created by

groups of visits. The coaddition process will take care

objects (eg., asteroids) from the templates. Difference

template given the time of observation, airmass, and seeing.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

3.5 Alerts

DIASources

3.5.1 Information Contained in Each Alert

For each detected

DIASource

, LSST will emit an “Event Alert” within

OTT1

visit (defined as the end of image readout from the LSST Camera).

VOEvent

format

, and should be readable

VOEvent

-compliant

clients.DMS-REQ-0002

Each alert

VOEvent

packet) will at least include the following:

•

alertID

: An ID uniquely identifying this alert. It can also

the Level 1 database as it existed when this alert

DMS-REQ-0274

was issued.

•

Level 1 database ID

•Science Data:

–

The

DIASource

record that triggered the alert

–

The entire

DIAObject

(or

SSObject

) record

–

Previous 12 months

DIASource

records

diaCharacterization-

Cutoff

–

Matching

Object

IDs from the latest Data Release, if they exist,

their

DIASource

records

•Cut-out of the difference image

DIASource

(10

centered

bytes/pixel,

on the

FITS MEF)

•Cut-out of the template image

DIASource

centered

(10 bytes/pixel,

on the

FITS MEF)

The variable-size cutouts will be sized so as to encompass

source, but be no

㐀㄀뜀㐀㄀

smaller

pixels.

than

The provided images will comprise

float), variance (32 bit float), and mask (16 bit flags) planes,

for further processing (e.g., WCS, zero-point, PSF, etc.).

The items above are meant

information

to represent

transmitted

the with each alert;

tent of the alert packet itself will

VOEvent

(or

be formatted

other relevant)

to confirm

dard. Where the existing standard is inadequate for LSST

and work with the community to reach a common solution.

Or some other format that is broadly accepted and used by the community

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

With each alert, we attempt to include as much

DIA-

information

Source

as possible, to minimize the need for follow-up database

sification and decision making at the user end, and relaxes

on the Project end.

3.5.2 Receiving and Filtering the Alerts

Alerts will be transmitted

VOEvent

format, using standard IVOA protocols

DMS-REQ-0002

Transport Protocol; VTP

). As a very high rate of alerts is expected,

꤄

10 million

per night, we plan for public VOEvent Event Brokers

to be the primary end-points

event streams. End-users will use these brokers to classify

their science goals.

not

End-users

able to subscribe

will

to full, unfiltered,

coming directly from LSST

To directly serve the end-users, LSST will provideDMS-REQ-0342

a basic,

This service will run at the LSST U.S. Archive Center (at

simple filters that limit what alerts are ultimately forwarded

. These

user defined

filters

will be possible to specify using an SQL-like declarative

(likely Python) code. For example, here’s what a filter may

# Keep

onln

yever

缄

before

缄

seee

nvents

within

two

# effective

radii

ofa galaxy

.Thii

ssforillustration

# only

;th exact

methods

/members

/APImsaychange

def filter

( alert ):

if len

( alert .sources) > 1:

return

False

nn = alert . diaobject . nearest_neighbors [0]

if not

nn. flags .GALAXY:

return

False

VOEvent Transport Protocol is currently an IVOA Note, but we understand

bring it up to full IVOA Recommendation status.

These brokers are envisioned to be operated as a public service by third

with LSST.

This is due to finite network bandwidth available: for example, 100

꤄

100Mbps

stream (the peak full stream data rate at end of the first year of operations)

from the archive center, just to serve the alerts.

More specifically, to their VTP clients. Typically, a user will use

LSST Archive Center) to set up the filters, and use

VOEvent

their

stream.

VTP client to

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

return

nn. dist < 2. * nn.Re

We emphasize that this LSST-provided

not

capability

intended to

will

satisfy

the wide variety of use cases that a full-fledged public Event

do not plan to provide

exclusive

classification

any

to a unique category of

the

SRD

specification, however, we will provide a limited number

small number of object types of common interest. These

DMS-REQ-0348

will

such as “is the light curve consistent with an RR Lyra?”, and

ping selections, designed to provide good completeness but

No information beyond what

VOEvent

is contained

packet will

inbe

the

available to

defined or user-defined filters (e.g., no cross-matches to

run time of user defined filters will be limited by available

not be guaranteed.VOEvents

The number

transmitted

to each user per visit

as well (e.g., the equivalent of 20 full-size alert

numBrokerAlerts

packets

DMS-REQ-0343

tled depending on load). Finally, the total number of simultaneous

limited – in case of overwhelming interest, a TAC-like

numBrokerUsers

proposal

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

4 Level 2 Data Products

4.1 Overview

Level 2 data products result from direct image

analysis. They’re designed to

static

sky

science (eg., studies of galaxy evolution, or weak lensing),

is not time sensitive (eg. statistical investigations

ucts (reduced single-epoch

Processed

exposures,

Visit

or, sometimes,

called

Images

calibrated

exposures

, and coadds), and catalog products (tables of

DMS-REQ-0069

objects,

DMS-REQ-0279

DMS-REQ-0281

DMS-REQ-0275

DMS-REQ-0267

erties, and related metadata).

Similarly to Level

DIAObjects

1 catalogs

and

DIASources

Objects

in the Level 2 catalog

DMS-REQ-0276

DMS-REQ-0275

resent the astrophysical phenomena (stars,

Sources

represent

galaxies, quasars,

their single-epoch

Sources

observations.

are independently detected and measured

epoch exposures and

Source

recorded

table.

in the

DMS-REQ-0267

The master

Objects

list

inof

Level 2 will be generated by associating

single-epoch

DIASource

detections and the lists of sources

CoaddSources

detected

DMS-REQ-0275

We plan to build coadds designed

“deep

coadds”

maximize

) and coadds

depth

designed

DMS-REQ-0279

(

to achieve a good combination

“best

of depth

seeing

and

unless

coadds”

seeing

algorithms

(

will enable these two to be the same.

short-period

We will

(eg.

also

yearly,

build

DMS-REQ-0330

or multi-year) coadds. The flux limit in deep coadds

DMS-REQ-0337

will

dividual visits, and the best seeing coadds will help with

The short-period coadds are necessary to avoid missing faint

ability. These coadds will be built for all bands, as well

(

“multi-color

Not

coadds”

all of these will

after

be preserved

sources are detected

DMS-REQ-0281

sured (see

4.4.3

for

§ details). We will provide a facility to regenerate

3 tasks

5).(§

DMS-REQ-0311

DMS-REQ-0275

The deblender will be run simultaneously on the catalog

detected in the coadds,

the

DIAObject

catalog from the Level 1 database, and one or more

use the knowledge of peak positions, bands, time, time variability

As opposed to

difference

, for

image

Level 1.

The source detection algorithm we plan to employ finds regions of connected

먉　딉

threshold

PSF-likelihood

in the

of the

image

visit (or coadd). These

footprints

.regions

Each footprint

are called

may

have one or

peaks

, and it is these peaks that the deblender will use to infer

objects blended in each footprint.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

epoch

Source

detections), inferred motion, Galactic longitude

information to produce a master

Objects

.list

Metadata

of deblended

on why and how

particular

Object

was deblended will be kept.

DMS-REQ-0276

The properties

Objects

, including

their exact positions, motions,

will be characterized by MultiFit-type algorithms

Finally, to enable studies of

Objects

variability,

will be measured

the fluxes

on individual

of all

visits (using both direct and difference images), with their

resolutions kept constant.

forced

This process

photometry

(see

4.2.4

§),

known

and the

flux measurements will

ForcedSource

be stored

table.

in the

DMS-REQ-0268

4.2 Level 2 Data Processing

Figures

3and

4present a high-level overview of the Level 2 data

. Log-

ically

, the processing begins with single-visit image reduction

followed by global astrometric and photometric calibration,

coadds, association and deblending, object characterization,

surements.

The following is a high-level description of steps which

processing (bullets 1 and 2 below

Single

map

Visit

to,pipeline

Processing

Figure

3, bullet

3 is pipeline

Image

Coaddition

, bullets 4-6 map

Coadded

to pipeline

Image

and

Analysis

bullet 7 is pipeline

Multi-epoch

Object

Characterization

Single Visit

: RawexposuresarereducedtoProcessedVisit

Processing

Sources

are independently detected, deblended, and measured

ments (instrumental fluxes and

Source

shapes)

table.

are stored

DMS-REQ-0267

in the

Relative calibration

: The survey is internally calibrated, both

metrically. Relative zero-point and astrometric

DMS-REQ-0029

corrections

Sufficient data is kept to reconstruct the normalized

猌

쌉

⤀怌⨀

(see

DMS-REQ-0030

system

Eq.

SRD

5,) at every position in the focal plane at the time

§ 3.3.4

SRD

of.the

“MultiFit algorithms” are those that fit a PSF-convolved model (so-called

observations of an object. This approach is in contrast to measurement

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Coadd creation

: Deep, seeing optimized, and short-period

DMS-REQ-0279

per-band

DMS-REQ-0337

픉젉툉줉��

bands, as well as deeper, multi-color, coadds

. Transient sources (including

DMS-REQ-0281

Solar System objects, explosive transients, etc), will

§4.4.3

for details.

Coadd source

detection

Sources will be detected on all coadds generated

step. The source detection algorithm will detect

DMS-REQ-0277

regions

footprints

, above

the먉　딉

nominal

threshold

PSF-likelihood

in the

of the

image

visit. An

appropriate algorithm will be run to also detect extended

(eg., binned detection algorithm from SDSS).

peaks

Each

, DMS-REQ-0349

footprint

and the collection of these peaks (and their membership

of this stage.

Association and

. The

deblending

next stage in the pipeline,DMS-REQ-0275

which

just

the

call

deblender

, will synthesize a list of unique objects.

the catalogs

CoaddSources

of, catalogs

DIASources

of,

DIAObjects

and

SSObjects

detected

on difference images, and objects from external catalogs

DMS-REQ-0034

The deblender will make use of all information available

knowledge of peak positions, bands, time, time variability

gitude and latitude, etc. The output of this

Objects

stage

is a

Coadd object characterization

. Object properties such as adaptive moments

fluxes will be measured on the coadds. These will be stored

Object

table. Models fit in multi-epoch object characterization

coadds and stored.

DMS-REQ-0276

Multi-epoch object

. A

characterization

set of predefined model fits and measurements

be performed on

Objects

eachidentified

of the

in the

taking

all

step,

available

multi-epoch data

. Model

into

fits

account

will be performed

MultiFit

-type algorithms.

using

are coadded first, and the properties are measured from the coadded pixels.

Note that some LSST documents refer to

Data Release

Processing

which includes both Level 1 reprocessing

(see

3.3.5

§), and the Level 2 processing described here.

The actual implementation may parallelize these steps as much as possible.

We’ll denote the “band” of the multi-color coadd as ’M’.

Note that

Sources

are not considered when generating the

Object

list (given the large number of

each band, the false positives close to the faint end would increase the

algorithms). It is possible for intermittent sources that are detected

visits to be undetected in coaddds, and

Object

thus

To enable

to not easy

have identification

matching

Sources

, the nearest

Object

associatedSource

with

, if

each

any, will be recorded.

Depending on the exact implementation of the deblender, this stage

deblended footprints and pixel-weight

Object

record.

maps) to each deblended

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Rather than coadding a set of images and measuring object

MultiFit simultaneously fits PSF-convolved models to

This reduces systematic errors,

먉　딉

, and

improves

allows for

the fitting

overall

dependent quantities degenerate with shape on the coadds

motion). The models we

not

allow

plan to

for

fitflux

will

variability

DMS-REQ-0275

(see

Forced Photometry

. Source fluxes will be measured on every visit,

motion, shape, and the deblending parameters characterized

fixed. Measurements will be performed on both direct images

This process

forced

photometry

, will result in the characterization

each object in the survey. The

ForcedSource

fluxes will

table.

be stored

DMS-REQ-0268

in the

4.2.1 Object Characterization Measures

Properties of detected objects will be measured as a part

described in the previous section

Object

table.

and stored

These measurements

in the

designed to enable LSST “static sky” science. This section

erties will be measured and how those measurements will

of quantities being fit/measured,

4.3.1 see the table in §

All measurements discussed in this section

objects

, anddeal

willwith

be per-

properties

formed on multi-epoch coadds, or by simultaneously fitting

sources in individual visits, independent

4.2.3of all others,

To enable science cases depending on observations of non-variable

sky, we plan to measure the following using the MultiFit

•

Point source

. The

model

observed

fit

object is modeled as a point source

motion and parallax and constant flux (allowed to

DMS-REQ-0276

be different

is a good description for non-variable stars and other

ters will be simultaneously constrained using information

in all

bands

•

Bulge-disk

. The

model

object

fit

is modeled as a sum of

츉 㸀㔀

a de

) DMS-REQ-0276

Vaucouleurs

and an exponential

츉 㸀㈀

) component.

(Sersic

This model is intended to

The fitting procedure will account for differential chromatic refraction.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

description of galaxies

. The object is assumed

. The

not

components

to move

the same ellipticity and center. The model is independently

total of 8 free parameters, which will be simultaneously

from all available epochs for each band. Where there’s

the likelihood (eg., small, poorly resolved, galaxies,

adopted to limit the range of its sampling.

We will also explore fitting the galaxy model simultaneously

parameters constrained to be the same or related via a hierarchical

As this reduces the number of overall model parameters

consider freeing up other parameters. One likely scenario

bulge and disk ellipticities to differ; another possibility

one or both components to vary. The ultimate determination

be driven by empirical tests of the robustness and quality

high-redshift galaxies.

DMS-REQ-0333

In addition to the maximum likelihood values of fitted

matrix, a number (currently

꤄

200, on

planned

average

) of independent

to be

samples

from the likelihood function will be provided. These will

partures from the Gaussian approximation, with shear

use case. A permissible descope, in case of insufficient

the posterior

픉

and

��

bands.

for

•

Standard

colors

Colors of the object in “standard seeing”

DMS-REQ-0276

(for

expected

survey줉

seeing

band,

꤄

0.9in

arcsec)

the

will be measured. These

guaranteed to be seeing-insensitive, suitable for estimation

•

Centroids

. Centroids will be computed independently for each

similar to that employed by SDSS. Information from all

epochs will be used to derive

the estimate. These centroids will be used for adaptive

dard color, and aperture measurements.

DMS-REQ-0276

•

Adaptive

moments

Adaptive moments will be computed using information

We may reconsider this choice if a better suited parametrization is

I.e., have zero proper motion and trigonometric parallax.

This choice of the number of independent samples will be verified during

The problem of optimal determination of photometric redshift is the

proach we’re taking here is conservative, following contemporary practices.

revisit the issue.

Whenever we

all

,say

it should be understood that this does not preclude reasonable

exclude data that would otherwise degrade the measurement.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

independently for each band. The moments of the PSF realized

ject will be provided as well.

DMS-REQ-0276

•

Petrosian and

Kron

Petrosian

fluxes

and Kron radii and fluxes will

dard seeing using self-similar elliptical apertures

The apertures will be

homogenized

PSF-corrected

, convolved

andto a canonical

PSF

. The radii will be computed independently for each

in each band, by integrating the

the

light

radius

within

measured

some multiple

canonical

(most

band

likely the

줉

band). Radii enclosing 50% and 90%

provided.

DMS-REQ-0276

•

Aperture surface

. Aperture

brightness

surface brightness will be computed

number

of concentric, logarithmically spaced, PSF-homogenized,

in standard seeing.

DMS-REQ-0276

•

Variability characterization

. Parameters will be provided, designed

odic and aperiodic variability

], in each bandpass.

features [We caution that

features in use when LSST begins operations are likely

baseline described here; this is to be expected given

domain astronomy.

number

However,

is unlikely

their to grow beyond the present

mate.

DMS-REQ-0276

4.2.2 Supporting Science Cases Requiring Full Posteriors

Science cases sensitive to systematics, departures of likelihood

ing user-specified priors, demand knowledge of the shape

a simple Gaussian approximation around the ML value. The

parameters and the estimate of photometric redshifts are

the full posterior is likely to be needed for LSST science

DMS-REQ-0046

DMS-REQ-0276

We currently plan to provide this information in two ways:

This is an attempt to derive a definition of elliptical apertures that

for a large galaxy, the correction to standard seeing will introduce

apertures for small galaxies will tend to be circular (due to smearing

method results in derived apertures that are relatively seeing-independent.

apertures

; the measured flux will still be seeing dependent and it is up to the

The shape of the aperture in all bands will be set by the profile of the

This procedure ensures that the color measured by comparing the flux in

consistent

aperture.

http://www.sdss.org/dr7/algorithms/photometry.html

See

for details.

The number will depend on the size of the source.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

ples from the likelihood function (in the case of the bulge-disk

parametric estimates of the likelihood function (for the

shown in5Table

, the

current꤄

allocation

200 samplesis

(on average) for the bulge-disk

and

꤄

100 parameters for describing the photo-Z likelihood

DMS-REQ-0333

The methods of storing likelihood functions (or samples

oped and optimized throughout Construction and Commissioning.

amount of data needed to be stored, can be overcome by compression

ample, simply noticing

꤄

0.5%

thataccuracy

not more than

is needed for sample

one to increase the number of

㔀

samples

More advanced

by a factor

techniques,

PCA analysis of the likelihoods across the entire catalog,

providing a better estimate of the shape of the likelihood

sented in

5should

Table be thought

conservative

of as a , which

estimate

we plan to improve

upon as development continues in Construction.

4.2.3 Source Characterization

Sources will be detected on individual visits as well as the

will primarily serve as inputs to the construction

4.2

, of the

and may support other LSST science cases as seen fit by the

objects whose shapes vary over time).

DMS-REQ-0277

DMS-REQ-0267

The following

Source

properties are planned to be measured:

•

Static point source

. The source

model fit

is modeled as a static point

total of 3 free

嘌

parameters

夌

, flux). This

( model is a good description

unresolved sources.

DMS-REQ-0267

•

Centroids

. Centroids are currently planned to be computed using

that employed by SDSS. These centroids will be used for

magnitude measurements.

DMS-REQ-0267

•

Adaptive

moments

Adaptive moments will be computed. The moments

ized at the position of the object will be provided

DMS-REQ-0267

as well.

•

Aperture surface

. Aperture

brightness

surface brightness will be computed

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

number

of concentric, logarithmically spaced, PSF-homogenized,

DMS-REQ-0267

Note that

not

plan

do to fit extended source Bulge+Disk

Sources

, nor

models to

measure per-visit Petrosian or Kron fluxes. These are object

to vary in

, and

time

will be better characterized by MultiFit (in

Object

table). For example,

although a simple extendedness characterization is present

separation (an estimate of the probability that a source

better characterized by MultiFit.

4.2.4 Forced Photometry

DMS-REQ-0268

DMS-REQ-0287

Forced Photometry

is the measurement of flux in individual visits,

and the deblending parameters of an object. It enables the

object’s flux, irrespective of whether the flux in any given

single-visit detection threshold.

Forced photometry will be performed

Objects

, using

on allboth

visits,

direct

forimages

all

and difference images. The measured

ForcedSources

fluxes will

table.

be stored

Due to in

space constraints, we only plan to measure the PSF flux.

4.2.5 Crowded Field Photometry

A fraction of LSST imaging will cover areas of high object

the Galactic plane, the Large and Small Magellanic Clouds,

(among others).

LSST image processing and measurement software, although

in non-crowded regions, is expected to perform well in areas

applications development plan envisions making the deblender

and latitude, and permitting it to use that information

blend objects. While not guaranteed to reach the accuracy

crowded field photometry codes, we expect this approach will

The number will depend on the size of the source.

Objects

that

change shape with time would, obviously, be of particular

in the

Source

table should suffice to detect these. Further per-visit shape characterization

Level 3 task.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

in areas of moderately high crowding.

Note that this discussion only

direct

pertains

images

Crowding

to processing

is not expected

to significantly impact the quality

difference

of data products

(i.e.,

images

Level

derived

1).

4.3 The Level 2 Catalogs

This section presents the contents of key Level 2 catalog

(see

3.3

), here we present

conceptual

the for

schemas

the most important Level 2

Object

Source

, and

ForcedSource

tables). The tables themselves

LDM-153

. are defined

These convey

what

data will be recorded in each table,

how

. rather

For

than

example, columns whose type

radec

)is

may

anbe

array

expanded

(eg.,to one table

per element of the

decl

array

) once

(eg.,

this schema is translated to SQL.

tables to be presented are normalized (i.e., contain no redundant

since the band of observation

Source

cantable

be found

to the

by joining

table with

a exposure

metadata, there’s no

band

column

in the

Source

named

table. In the as-built database,

views presented to the users will be appropriately

DMS-REQ-0332

denormalized

4.3.1

Object

TheTable

DMS-REQ-0275

Table 5: Level

Object

2 Catalog

Table

Name

Type

Unit

Description

objectId

uint64

Unique object identifier

parentObjectId

uint64 ID of theObject

parent

this object has

been deblended from, if any.

radec

double[6][2] arcsec

Position of the object (centroid),

puted independently in each

The centroid will be computed

an algorithm similar to that

by SDSS.

radecErr

double[6][2] arcsec

Uncertainty

radec

. of

Continued on next page

The SQL definition itself can be found in the

cat

package.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table 5: Level

Object

2 Catalog

Table

Name

Type

Unit

Description

psRadecTaidouble time

Point source model: Time at which

object was at

psRadec

position

psRadec

double[2]

degrees

Point source

⤀嘌ⴀ夌⨀

model:

position of

the object

psRadecTai

at time

psPm

float[2]mas/yr Point source model: Proper

vector.

psParallaxfloat

mas

Point source model: Parallax.

psFlux

float[ugrizy] nmgy

Point source model fluxes

psCov

float[66]

variousPoint-source model covariance

trix

psLnL

float

Natural

찉켉젉

likelihood of the observed

data given the point source model.

psChi2

float

氌

㌀

statistic of the model fit.

psNdata

int

The number of data points (pixels)

used to fit the model.

bdRadec

double[2][ugrizy]

B+D model

⤀嘌ⴀ 夌⨀

position of the ob-

ject, in each band.

bdEllip

float[2][ugrizy]B+D model: Ellipticity

⤀옉

㈀

ⴀ 옉

㌀

⨀

of the ob-

ject.

bdFluxB

float[ugrizy] nmgy

B+D model: Integrated flux of

Vaucouleurs component.

bdFluxD

float[ugrizy] nmgy

B+D model: Integrated flux of

ponential component.

bdReB

float[ugrizy] arcsec

B+D model: Effective radius of

Vaucouleurs profile component.

bdReD

float[ugrizy] arcsec

B+D model: Effective radius of

ponential profile component.

Continued on next page

Point source model assumes that fluxes are constant in each band. If

psFlux

will effec-

tively be some estimate of the average flux.

Not all elements of the covariance matrix need to be stored with same

stored as

32꤄

bit

seven

floats

significant

(

digits), the

꤄covariances

three significant

may

꤄

1% bedigits

stored

Though we refer to this model as “Bulge plus Disk”, we caution the

physically motivated, should not be taken too literally.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table 5: Level

Object

2 Catalog

Table

Name

Type

Unit

Description

bdCov

float[36][ugrizy]

B+D model covariance matrix

bdLnL

float[ugrizy]

Natural

찉켉젉

likelihood of the observed

data given the bulge+disk model.

bdChi2

float[ugrizy]

氌

㌀

statistic of the model fit.

bdNdata

int[ugrizy]

The number of data points (pixels)

used to fit the model.

bdSamples float16[9][200][ugrizy]

Independent samples of bulge+disk

likelihood surface. All sampled

ties will be stored

꤄

3 sig-

with at least

nificant digits of precision.

ber of samples will vary from

to object, depending on how well

object’s likelihood function

mated by a Gaussian.

stdColor

float[5]mag

Color of the object measured

dard seeing”. While the exact

rithm is yet to be determined,

color is guaranteed to be seeing-

independent and suitable for

determinations.

stdColorErr

float[5]mag

Uncertainty

stdColor

.of

Ixx

float

asec

㌀

Adaptive second moment of

source intensity. See Bernstein

Jarvis

2] for

[ detailed discussion

all adaptive-moment related

ties

Iyy

float

asec

㌀

Adaptive second moment of

source intensity.

Ixy

float

asec

㌀

Adaptive second moment of

source intensity.

Icov

float[6]asec

㔀

Ixx

Iyy

Ixy

covariance matrix.

Continued on next page

See

psCov

for notes on precision of variances/covariances.

http://ls.st/5f4

for a brief summary.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table 5: Level

Object

2 Catalog

Table

Name

Type

Unit

Description

IxxPSF

float

asec

㌀

Adaptive second moment for the

IyyPSF

float

asec

㌀

Adaptive second moment for the

IxyPSF

float

asec

㌀

Adaptive second moment for the

float[ugrizy]

Fourth order adaptive moment.

petroRad

float[ugrizy] arcsec

Petrosian radius, computed

liptical apertures defined by

tive moments.

petroRadErr

float[ugrizy] arcsec

Uncertainty

petroRad

petroBand int8

The band of the

petroRad

canonical

petroFlux float[ugrizy] nmgy

Petrosian flux within a defined

ple of the canonical

petroRad

petroFluxErr

float[ugrizy] nmgy

Uncertainty

petroFlux

petroRad50

float[ugrizy] arcsec

Radius containing 50% of Petrosian

flux.

petroRad50Err

float[ugrizy]

Uncertainty

arcsec

petroRad50

petroRad90float[ugrizy] arcsec

Radius containing 90% of Petrosian

flux.

petroRad90Err

float[ugrizy]

Uncertainty

arcsec

petroRad90

kronRad

float[ugrizy] arcsec

Kron radius (computed using elliptical

apertures defined by the adaptive

ments)

kronRadErrfloat[ugrizy] arcsec

Uncertainty

kronRad

kronBand

int8

The band of the

kronRad

canonical

kronFlux

float[ugrizy] nmgy

Kron flux within a defined multiple

the canonical

kronRad

kronFluxErr

float[ugrizy] nmgy

Uncertainty

kronFlux

kronRad50 float[ugrizy] arcsec

Radius containing 50% of Kron

kronRad50Err

float[ugrizy] arcsec

Uncertainty

kronRad50

kronRad90 float[ugrizy] arcsec

Radius containing 90% of Kron

kronRad90Err

float[ugrizy] arcsec

Uncertainty

kronRad90

apNann

int8

Number of elliptical annuli

low).

Continued on next page

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table 5: Level

Object

2 Catalog

Table

Name

Type

Unit

Description

apMeanSb

float[6][

apNann

]nmgy/as

㌀

Mean surface brightness within

nulus

apMeanSbSigma

apNann

float[6][

]nmgy/as

㌀

Standard deviation

apMeanSb

extendedness

float

A measure of extendedness,

puted using a combination of

able moments, or from a likelihood

tio of point/B+D source models

algorithm

옉��퐉옉츉씉옉씉츉옉팉팉 㸀㈀

TBD).

im-

plies a high degree of confidence

the source is

옉��퐉옉츉씉옉씉츉옉팉팉 㸀

extended.

㄀

implies a high degree of confidence

that the source is point-like.

lcPeriodicfloat[6

뜀

32]

Periodic features extracted

ference image-based light-curves

ing generalized Lomb-Scargle

odogram [Table

lcNonPeriodic

float[6

뜀

20]

Non-periodic features extracted

difference image-based light-curves

[Table

photoZ

float[2

뜀

100]

Photometric redshift likelihood

ples–pairsof

��

찉켉젉댉

) – computed

(

us-

ing a to-be-determined published

widely accepted algorithm at

of LSST Commissioning.

flags

bit[128]

Various useful flags.

4.3.2

Source

Table

Source

measurements are performed independently on individual

enable relative astrometric and photometric calibration,

A database function will be provided to compute the area of each annulus,

aperture flux.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

noise objects, and studies of high SNR objects that

DMS-REQ-0267

vary in

Table 6: Level

Source

2 Catalog

Table

Name

Type

Unit

Description

sourceId

uint64

Unique source

identifier

ccdVisitIduint64

ID of CCD and visit where this

was measured

objectId

uint64

ID ofObject

thethis source was asso-

ciated with, if any.

ssObjectIduint64

ID ofSSObject

the this source has been

linked to, if any.

parentSourceId

uint64 ID of theSource

parent

this source has

been deblended from, if any.

float[2]pixels Position of the object (centroid),

puted using an algorithm similar

that used by SDSS.

xyCov

float[3]

Covariance matrix

for

radec

double[2]

arcsec

Calibrated

嘌

夌

) of(the source, trans-

formed

from

radecCov

float[3]arcsec Covariance matrix

radec

for

apFlux

float

nmgy

Calibrated aperture flux.

apFluxErr float

nmgy

Estimated uncertainty

apFlux

sky

float

nmgy/asec

㌀

Estimated background (sky)

brightness at the position (centroid)

the source.

skyErr

float

nmgy/asec

㌀

Estimated uncertainty of

sky

psRadec

double[2]

degrees

Point source

⤀嘌ⴀ夌⨀

model:

position of

the object.

psFlux

float

nmgy

Calibrated point source model

psCov

float[6]variousPoint-source model covariance

trix

Continued on next page

It would be optimal if the source ID is globally unique across all releases.

on technological and space constraints.

Not all elements of the covariance matrix will be stored with same precision.

as 32 bit

꤄

floats

seven(significant digits), the

꤄covariances

three significant

may

꤄

1% be

).digits

stored(to

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table 6: Level

Source

2 Catalog

Table

Name

Type

Unit

Description

psLnL

float

Natural

찉켉젉

likelihood of the observed

data given the point source model.

psChi2

float

氌

㌀

statistic of the model fit.

psNdata

int

The number of data points (pixels)

used to fit the model.

Ixx

float

asec

㌀

Adaptive second moment of

source intensity. See Bernstein

Jarvis

2] for

[ detailed discussion

all adaptive-moment related

ties

Iyy

float

asec

㌀

Adaptive second moment of

source intensity.

Ixy

float

asec

㌀

Adaptive second moment of

source intensity.

Icov

float[6]asec

㔀

Ixx

Iyy

Ixy

covariance matrix.

IxxPSF

float

asec

㌀

Adaptive second moment for the

IyyPSF

float

asec

㌀

Adaptive second moment for the

IxyPSF

float

asec

㌀

Adaptive second moment for the

apNann

int8

Number of elliptical annuli

low).

apMeanSb

float[

apNann

] nmgy

Mean surface brightness within

nulus.

apMeanSbSigma

apNann

float[

] nmgy

Standard deviation

apMeanSb

extendedness

float

A measure of extendedness,

puted using a combination of

able moments (exact algorithm

옉��퐉옉츉씉옉씉츉옉팉팉 㸀㈀

implies a high de-

gree of confidence that the source

extended.

옉��퐉옉츉씉옉씉츉옉팉팉 㸀㄀

implies

a high degree of confidence that

source is point-like.

Continued on next page

http://ls.st/5f4

for a brief summary.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Table 6: Level

Source

2 Catalog

Table

Name

Type

Unit

Description

flags

bit[64]bit

Various useful flags.

4.3.3

ForcedSource

Table

DMS-REQ-0268

Table 7: Level

ForcedSource

2 Catalog

Table

Name

Type

Unit

Description

objectId

uint64

Unique object identifier

ccdVisitIduint64

ID of CCD and visit where this

was measured

psFlux

float

nmgy

Point source model flux on direct

age, if performed.

psFluxErr

float

nmgy

Point source model flux error,

to 1% precision.

psDiffFlux float

nmgy

Point source model flux on difference

image, if performed.

psDiffFluxErr

float

nmgy

Point source model flux error,

to 1% precision.

flags

bit[8] bit

Various useful flags.

4.4 Level 2 Image Products

4.4.1 Visit Images

DMS-REQ-0065

Raw exposures, including individual snaps, and Processed

for download as FITS files. They will be downloadable both

User Interface, as well as using machine-friendly APIs.

Required calibration data, processing metadata,DMS-REQ-0130

and all

DMS-REQ-0298

will be provided to enable the user to generate bitwise id

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

images

DMS-REQ-0308

4.4.2 Calibration Data

All calibration frames (darks, flats, biases, fringe,

DMS-REQ-0130

etc.)

for download as FITS files.

DMS-REQ-0065

DMS-REQ-0298

All auxiliary telescope data, both raw (images with spectra)

derived atmosphere models), will be preserved and made available

4.4.3 Coadded Images

In course of Level 2 processing, multiple classes and numerous

•A set

deep

coadds

. One deep coadd will be created

픉젉툉줉��

bands,

for each

DMS-REQ-0279

plusof

a seventh, deeper, multi-color coadd. These coadds

DMS-REQ-0281

will

combination of depth (i.e., employ no PSF matching) and

nificantly degraded seeing may be omitted). Transient

objects, explosive transients, etc), will be removed.

astrophysical

backgrounds

The six per-band coadds will be kept indefinitely

Their

DMS-REQ-0334

and made

primary purpose is to enable the end-users to apply alternative

rithms, perform studies of diffuse

structures, and for

•A set

best

seeing

coadds. One deep coadd will be

픉젉툉줉��

created

bands,

for each

using only the best seeing data (for example, using only

seeing distribution). These will be built to assist the

will be kept indefinitely and made available to the

DMS-REQ-0330

users.

DMS-REQ-0334

•A set

short-period

coadds. These will comprise of multiple

DMS-REQ-0337

(ugrizyM)

multi-year coadds. Each of these sets will be created

and otherwise share the characteristics of the deep coadds

designed to enable detection of long-term variable or

objects that would

Assuming identically performing software and hardware configuration.

See

LDM-151

for more details.

For example, using “background matching” techniques [

For example, nearby high proper motion stars.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

“washed out” (or rejected)

not

full-depth

plan to keep

coadds.

and make

coadds available

. We will retain and provide sufficient metadata

them using Level 3 or other resources.

DMS-REQ-0298

DMS-REQ-0311

•One (ugrizyM) set of PSF-matched coadds. These

DMS-REQ-0335

will be

shapes of objects at “standard”

We do not plan

seeing.

to keep and make these

available

. We will retain and provide sufficient metadata for

using Level 3 or other resources.

The exact details of which coadds to build, and which ones

struction without affecting the processing system design

(raw image input and warping) are constant in the number

management system

sensitive

design to

number

the total

and

size

coadds

kept

to–be

these are the relevant constraining variables.

We reiterate

not all

that

coadds will be kept and served to the public

, though sufficient

metadata will be provided to users to recreate them on their

entirely “virtual”: for example, the PSF-matched coadds

volutions of postage stamps when the colors are measured.

DMS-REQ-0311

We will retain smaller sections of all generated coadds,

targeted science. Retained sections may be positioned

DMS-REQ-0338

interest such as overlaps with other surveys, nearby galaxies,

4.5 Data Release Availability and Retention Policies

Over 10 years of operations, LSST will produce eleven data

survey operations, and one every subsequent year. Each data

ing of all data from the start of the survey, up to the cutoff

The contents of data releases are expected

꤄

70PB

to range

for DR11

from

(this includes the raw images, retained coadds, and catalogs).

to keep all data releases loaded and accessible at all times.

The coadds are a major cost driver for storage. LSST Data Management

serve seven

ugrizyM

coadds,

over the full footprint of the survey.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

Instead,

only the contents of the most recent data release, and

will be kept on fast storage and with catalogs

. Statistics

loaded

DMS-REQ-0313

collected

into

by prior surveys (eg., SDSS) show that users nearly always

data release, but sometimes may use the penultimate one (this

publication of a new data release). Older releases are used

To assist with data quality monitoring

small, overlapping,

and assessment

samples

from older releases will be kept

. The

loaded

sample

size

the database

is expected

DMS-REQ-0339

order

꤄

1-5%

of the data release data, with larger samples kept

goal is to allow one to test how the reported characterization

release to release.

Older releases will be archived to

will

mass

not

storage

be able

(tape).

to perform

The

databasequeriesagainstarchivedreleases

. They will be made available as bulk

some common format (for example, FITS binary tables). Database

scripts will be provided for users who wish to set up a running

data release database on their systems.

DMS-REQ-0077

DMS-REQ-0300

All raw data used to generate any public data product (raw

telemetry, configuration metadata, etc.) will be kept

DMS-REQ-0309

and

DMS-REQ-0346

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

5 Level 3 Data Products and Capabilities

Level 3 capabilities

are envisioned to enable science cases that would

co-location of user processing and/or data within the LSST

requirement for Level 3 is

established

SRD

in § 3.5 of the

OSS-REQ-0140

LSST

Level 3 capabilities include three separate deliverables:

1.Level 3 Data Products and associated storage resources

2.Level 3 processing resources, and

3.Level 3 programming environment and framework

Many scientists’ work may involve using two or all three of

be used independently. We describe each one of them in the

5.1 Level 3 Data Products and Associated Storage Resources

DMS-REQ-0290

These are data products that

are

any

generated

computing

resources

users

that are then brought to an LSST Data Access Center (DAC)

for these capabilities includes the physical storage and

to support them.

For catalog data products, there is an expectation that they

1 (L1) and Level 2 (L2) catalogs to enable analyses combining

that either the user-supplied tables include keys from

for key-equality-based joins with them (example: a table

galaxies, with a column of object IDs that can be joined to

are columns that can be used for spatial (or temporal, or analogous)

The latter implies that such L3 table’s columns must be in

units as the corresponding L1/L2 columns.

DMS-REQ-0123

DMS-REQ-0124

There is no requirement that Level 3 data products (L3DPs)

than that they be joinable with them. For instance, a user

that they might want to bring into federation with the LSST

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

of as a Level 3 Data Product as long as they have “LSST-ized”

coordinate, time, measurement systems, etc. Nevertheless,

L3DPs to be derived from processed LSST data.

There could also be L3 image data products; for example, user-generated

selection criteria or stacking algorithms (eg. the so-called

moving objects).

DMS-REQ-0127

Any L3DP may have access controls associated with it, restricting

to a list of people, to a named group of people, or allowing

DMS-REQ-0340

The storage resources for

SRD

L3DPs

requirement

come out of

for

the

10% of LSST data

agement capabilities to be devoted to user processing. In

trolled by some form of a “space allocation committee”. Users

baseline automatic allocation, beyond which a SAC proposal

into account scientific merit, length of time for which the

of the data to others, in setting its priorities. DMS-REQ-0119

It is to be decided whether users will be required to provide

behind their L3DPs, or whether the SAC may include the availability

mation in its prioritization. Obviously if a user intends

a group it will be more important that supporting information

Level 3 data products that are found to be generally useful

fairly complex process that ultimately involves the project

and running LSST-style code that implements the algorithm

product (it’s not just relabeling an existing L3DP as L2).

support for such migrations.

5.2 Level 3 Processing Resources

DMS-REQ-0119

These are project-owned computing resources located at

allocation to all users with LSST data rights. They may

involves the LSST data and advances LSST-related science.

computing resources is that they are located with excellent

catalog datasets at Level 1 and Level 2.

not

There

project-owned,

may be other

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

resources available at the LSST DACs

; their use is beyond the scope of this

to note that reasonable provisions

possible

will beto

made

useto

them

ensure

to process

large quantities of LSST data.

Level 3 processing resources will, at least, include systems

style processing, probably similarly configured to those

release production processing. It is to be determined whether

would be provided, such as large-memory machines; this is

need (or lack thereof) for such resources.

There will be a time allocation committee (TAC) for these

user may get a small default allocation (enough to run test

require a scientific justification. Priorities will be based

on whether the results of the processing will be released

specify what special flavors of computing will be needed (e.g.,

A fairly standard job control environment (like Condor),

permitted to work with it at a low, generic level. They will

levels of the LSST process control middleware

5.3

(but they may;

These processing resources can be available for use in any

It is not strictly required that they be used to process LSST

could be acceptable to run special types of cosmological

of an LSST analysis,

if the closeness to the data makes the LSST facility

work

. The TAC will take into account in its decisions whether

of the enhanced I/O bandwidth available to LSST data on these

5.3 Level 3 Programming Environment and Framework

DMS-REQ-0125

DMS-REQ-0128

As a part of the Level 3 Programming Environment and Framework,

able the LSST software stack to users, to aid in the analyses

code implementing the core processing algorithms (image

tion, building of coadds, image differencing, object detection,

object detection, etc.), the middleware necessary to run

For example, the U.S. DAC will be located at the National Petascale

Blue Waters supercomputer.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

the LSST database management system.

DMS-REQ-0033

DMS-REQ-0032

DMS-REQ-0308

These analyses could be done on LSST-owned systems (i.e.,

sources) but also on a variety of supported external systems.

personal Unix flavors (for example, common distributions

commonly used cluster and HPC environments. The vision is

ward use of major national systems such as XSEDE or Open

common commercial cloud environments. The decision of which

will be under configuration control and we will seek advice

cannot commit to too many flavors. In-kind contributions

ronments will be welcome and may provide a role for national

The Level 3 environment is intended, when put to fullest

productions-like runs on bulk image and/or catalog data,

tracking large groups of jobs in a batch system.

The Level 3 environment, in asymptopia, has a great deal

that the Project will use to build the Level 2 data releases.

it as a tool meant for the end-users imposes additional requirements:

•In order to be successful

user

computing

as aenvironment, it needs to

Experience with prior projects

has shown that if the production computing

ment is not envisioned from the start as being shared with

an experts-only tool that is too complicated, or too work-hardened,

•While it is desirable for the production computing to

resources, this option might not be exercised in practice

community, it’s a far more central capability. Early community

key to developing and maintaining these capabilities.

•Not all the capabilities of the LSST production environment

to the users. LSST-specific capabilities associated

stance, are not of interest to end-users.

For example, BaBar.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

5.4 Migration of Level 3 data products to Level 2

•For the migration to be considered, the creator of the

their data product public to the entire LSST data-rights

ing documentation and code. The code at first need not

even in an LSST-supported language.

•If the original proponent wrote her/his code in the C++/Python

(the “Level 3 environment”), it will be easier to migrate

using the same languages/frameworks does not guarantee

quality).

•If the original code was written in another language or

work, the project may consider rewriting it to required

•Taking on a new Level 2 DP means that the project is committing

data quality review, space allocation, and continuing

DR11.

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

6 Data Products for Special Programs

LSR-REQ-0075

LSST Survey

Specifications

SRD

, § 3.4)(LSST

specify that 90% of LSST observing

spend executing the so-called “universal cadence”. These

and 2 data products described earlier in this document.

The remaining 10% of observing time will be devoted to special

proved coverage of interesting regions of observational

very deep

툉 ꤄ ㌀㜀

,(per exposure) observations, observations

꤄

with

minute), and observations of “special” regions such as

Large and Small Magellanic Clouds. A third type of survey,

about 1% of the time, may also be considered.

The details of these special programs or micro surveys are

. Consequently,

the specifics of their data products are left undefined at

the

constraints

these data products, given the adopted Level

derstood that no special program will be selected that does

. This

allows us to size and construct the data management system,

inition of these programs this far in advance.

Processing for special programs will make use of the same

capabilities as the processing for universal cadence.

DMS-REQ-0320

The

more than

꤄

10% of computational and storage capacity of the LSST

When special products include time domain event alerts,

subject to the same latency requirements as LevelDMS-REQ-0321

data

DMS-REQ-0344

OTT1

For simplicity of use and consistency, the data products

in databases separate from the “main” (Level 1 and 2) databases.

allow for simple federation with Level 1/2/3 data

DMS-REQ-0322

products

As a concrete example, a data product complement for a “deep

pernova discovery and characterization may consist of: i)

encing the science images against a special deep drilling

iii) one or more “nightly co-adds” (co-adds built using

The initial complement is expected to be defined and selected no later

Or will come with additional, external, funding, capabilities, and/or

The contents of this document are subject to configuration control and may

their provisions waived without prior approval.

LARGE SYNOPTIC SURVEY TELESCOPE

DPDD

LSE-163

Latest Revision 2017-07-01

duced and made available

꤄

24 hours,

within

and iv) special deep templates,

L1PublicT

best recently acquired seeing data, produced on a fortnightly

Note that the data rights and access rules apply just as

example, while generated event alerts (if any) will be accessible

catalog products will be restricted to users with LSST data

References

[1]

[LDM-153]

, Becla,

LSST

J., 2013,

Database Baseline

,LDM-153,URLhttps://ls.st/

Schema

LDM-153

[2]Bernstein,G.M.,Jarvis,M.,2002,AJ,123,583arXiv:astro-ph/0107431),

doi:10.1086/338085,ADSLink

[3][LSE-29],Claver,C.F.,TheLSSTSystemsEngineeringIntegrated

SystemRequirements,LSE-29,URLhttps://ls.st/LSE-29

[4][LSE-30],Claver,C.F.,TheLSSTSystemsEngineeringIntegrated

SystemRequirements,LSE-30,URLhttps://ls.st/LSE-30

[5][LSE-61],Dubois-Felsmann,G.,2016,LSSTDataManagementSubsystemRequirements,

LSE-61,URLhttps://ls.st/LSE-61

[6][LPM-17],Ivezić,Ž.,TheLSSTScienceCollaboration,2011,LSSTScienceRequirements

ument,LPM-17,URLhttps://ls.st/LPM-17

[7]Ivezic,Z.,etal.,2008,ArXive-prints(arXiv:0805.2366),ADSLink

[8][LDM-133],Juric,M.,Lim,K.T.,Kantor,J.,2013,DataManagementUMLDomainModel,

LDM-133,URLhttps://ls.st/LDM-133

[9][DMTN-035],Juric,M.,Becker,A.,Shaw,R.,Krughoff,K.S.,Winter2013

LSSTDMDataChallengeReleaseNotes,DMTN-035,URLhttps://dmtn-035.lsst.io

LSSTDataManagementTechnicalNote

[10]Muinonen,K.,Belskaya,I.N.,Cellino,A.,etal.,

doi:10.1016/j.icarus.2010.04.003,ADSLink

[11][LDM-156],Myers,J.,Jones,L.,Axelrod,T.,2013,MovingObjectPipelineSystem,

LDM-156,URLhttps://ls.st/LDM-156

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

[12]Richards,J.W.,Starr,D.L.,Butler,N.R.,etal.,arXiv:1101.1959),

doi:10.1088/0004-637X/733/1/10,ADSLink

[13][LDM-151],Swinbank,J.D.,etal.,2017,DataManagementSciencePipelines,LDM-

151,URLhttps://ls.st/LDM-151

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

AAppendix:ConceptualPipelineDesign

Ahigh-levelconceptualoverviewoftheLSSTimageprocessing

inFigure2.Thepipelinedefinitionspresentedherearedriven

processingsteps;theydonotdescribeexactboundariesintheactualimplemention

cution,ordevelopmentresponsibilitieswithintheProject.

with1,2,and5-8isexecutedeverydaywhennewdataare

Products.AnnualDataReleaseprocessingincludespipelines

Alertproduction).ThesemainconceptualstepsinLSSTimage

ingpipelines(enumerationinthislistcorrespondstoenumeration2butnotethat

thesestepscanbeinterleavedintheactualprocessingflow):

1.SingleVisitProcessingpipeline(Figure3)producescalibratedandcharacterized

visitimagesfromrawsnaps.Themainprocessingsteps

tureremoval,backgroundestimation,sourcedetection,

pointspreadfunctionestimation,andastrometricand

2.ImageCoadditionpipeline(Figure4)producescoaddedimagesofdifferent

mizedfordepth,seeing,etc.)fromanensembleofsingle-visit

3.CoaddedImageAnalysispipeline(Figure4)definestheObjectlistandperformsinitial

measurementsoncoaddedimages.

4.Multi-epochObjectCharacterizationpipeline(Figure4)fitsalibraryofimagemodels

setofFootprintsofanObjectfamily,measuresadditionalquantities

facebrightnessinaseriesofannuli)notcapturedbythose

Photometry.Allthesemeasurementsareperformedon

differenceimages)andforallObjects.

5.ImageDifferencingpipeline(Figure5)producesdifferenceimagesfromasingle-visit

coadded(template)images.

6.DifferenceImageAnalysispipeline(Figure5)updatesDIAObjectandSSObjectlistswith

newDIASourcesdetectedonprocesseddifferenceimage,fitsalibrary

FootprintsoftheseDIASources,andforallDIAObjectsoverlappingthedifference

performsForcedPhotometryandrecomputessummaryquantities.

1processing,thispipelinealsoperformsForcedPhotometryDIAObjectson

differenceimagesfromthelast30days.precoveryWindow

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

Figure2:IllustrationoftheconceptualdesignofLSSTscience

ing.

7.AlertGenerationandDistributionpipeline(Figure5)usesupdatedDIAObjectsandDIA-

SourcestogenerateanddistributeAlerts(whichalsoincludepostagestamp

theDIASourceindifferenceimageandcoaddedtemplateimage).

8.MovingObjectProcessingpipeline(MOPS,Figure6))combinesallDIASourcesun-associated

intoplausibleSSObjectsandestimatestheirorbitalparameters.

stagesincludeassociatingnewDIASourceswithknownSSObjects,discoveringnewSS-

Objects,andorbitrefinementandmanagement.

Furtherdetailsaboutthepipelinedesignandimplementation

documentLDM-151.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

Figure3:IllustrationoftheconceptualalgorithmdesignforSingle

Figure4:IllustrationoftheconceptualalgorithmdesignforImage

ageAnalysis,andMulti-epochObjectCharacterizationpipelines.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

Figure5:IllustrationoftheconceptualalgorithmdesignforImage

ImageAnalysis,andAlertGenerationandDistributionpipelines.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.

LARGESYNOPTICSURVEYTELESCOPE

DPDDLSE-163LatestRevision2017-07-01

Figure6:Illustrationoftheconceptualalgorithmdesignforthe

Softwarepipeline.

Thecontentsofthisdocumentaresubjecttoconfigurationcontrolandmay

theirprovisionswaivedwithoutpriorapproval.