1. LargeSynopticSurveyTelescope(LSST)
  2. SciencePlatformDesign
  3. GregoryDubois-Felsmann,FrossieEconomou,andKian-TatLim
  4. LDM-542
  5. LatestRevision: 2017-07-07
  6. Abstract
  7. ChangeRecord
  8. Contents
      1. 1 Introduction 1
      2. 2 TheArchitectureoftheSciencePlatform 3
      3. 3 ThePortalAspect 11
      4. 4 TheNotebookAspect 24
      5. 5 TheAPI Aspect 27
      6. 6The InterconnectednessoftheSciencePlatform 31
      7. 7 ApplicationoftheSciencePlatforminsidetheProject 33
  9. SciencePlatformDesign
  10. 1 Introduction
      1. 1.1 ThePortalAspect
      2. 1.2 TheNotebookAspect
      3. 1.3 TheAPI Aspect
      4. 1.4 Architecture
  11. 2 TheArchitectureoftheSciencePlatform
      1. 2.1 DesignOverview
      2. 2.1.1 Functional Architecture
      3. 2.1.2 Deployment Architecture
      4. 2.2 DataAccessandStorage
      5. 2.2.1 Databases
      6. 2.2.2 Images
      7. 2.2.3 LSST-specificObjects
      8. 2.2.4 DataAccessServices
      9. 2.2.5 UserCatalogDataSupport
      10. 2.2.6 UserWorkspaceStorage
      11. 2.2.7 DataAccessPermissionsandQuotas
      12. 2.2.8 SupportofPreviousReleases
      13. 2.3 ComputingResources
      14. 2.3.1 Basicusercomputeservices
      15. 2.3.2 Large-scale batch and parallel computing
      16. 2.4 Authentication and Authorization
      17. 2.5 ResourceManagement
      18. 2.6 Cybersecurity Considerations
      19. 2.7 AdditionalSupportServices
  12. 3 The Portal Aspect
      1. 3.1 GenericDataBrowsing
      2. 3.2 DocumentationDelivery
      3. 3.2.1 Context-Dependent Help
      4. 3.2.2 User Guide
      5. 3.2.3 ReferenceandDeploymentManuals
      6. 3.3 CustomizedWorkflows
      7. 3.4 UseoftheLSSTStack
      8. 3.5 APIInteractionwiththePortal
      9. 3.6 Extensibility of Visualizations
      10. 3.8 Implementation
      11. 3.9 InterfacetotheAlertDistributionandFilteringServices
  13. 4 TheNotebookAspect
      1. 4.1 JupyterHub/JupyterLabservice
      2. 4.2 Pre-installedLSSTsoftware
      3. 4.3 Pre-configuredaccesstoLSSTdata
  14. 5 TheAPI Aspect
      1. 5.1 Overview-VOservicesandcustomLSSTservices
      2. 5.2 Catalogandothertabulardataaccess
      3. 5.2.1 TAPservice
      4. 5.2.2 ADQLimplementation
      5. 5.2.3 Returndataformats
      6. 5.3 Image data access
      7. 5.3.1 Image finding
      8. 5.3.2 Image etrieval
      9. 5.4 Metadataaccess
      10. 5.4.1 DataReleases
      11. 5.4.2 Tables
      12. 5.4.3 TableStructure
      13. 5.5 UserWorkspaceaccess
  15. 6 The InterconnectednessoftheSciencePlatform
      1. 6.1 Sharing Data
      2. 6.2 SharingQueries
      3. 6.3 Sharing State
  16. 7 ApplicationoftheSciencePlatforminsidetheProject
      1. 7.1 DataManagementTeams
      2. 7.2 CommissioningTeam
      3. 7.3 ScienceValidationTeam
      4. 7.4 ObservatoryOperations
  17. References

Draft
LargeSynopticSurveyTelescope(LSST)

Back to top


SciencePlatformDesign

Back to top


GregoryDubois-Felsmann,FrossieEconomou,andKian-TatLim

Back to top


LDM-542

Back to top


LatestRevision: 2017-07-07
DraftRevisionNOTYETApproved
– ThisLSSTdocumenthasbeenapprovedasaContent-Controlled
Document by the LSST DM Technical Control Team. If this document is changed or superseded,
the new document will retain the Handle designation shown above. The control is on the most
recent digital document with this Handle in the LSST digital archive and not printed versions.
Additional information may be found in the corresponding DM RFC. –
Draft Revision NOT YET
Approved

Back to top


Abstract
ThisdocumentdescribesthedesignoftheLSSTSciencePlatform,theprimaryuser-
facinginterfaceoftheLSSTDataManagementSystem.
LARGESYNOPTICSURVEYTELESCOPE

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07

Back to top


ChangeRecord
Version Date
Description
Ownername
1
2017-07-07 FirstDraft.
Gregory Dubois-
Felsmann, K-T Lim,
FrossieEconomou
Document source location:
https://github.com/lsst/LDM-542
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
ii

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07

Back to top


Contents
1 Introduction
1
1.1 ThePortalAspect ...................................... 1
1.2 TheNotebookAspect .................................... 2
1.3 TheAPI Aspect ........................................ 2
1.4 Architecture ......................................... 2
2 TheArchitectureoftheSciencePlatform
3
2.1 DesignOverview ...................................... 4
2.1.1 FunctionalArchitecture ............................... 4
2.1.2 DeploymentArchitecture .............................. 4
2.2 DataAccessandStorage .................................. 5
2.2.1 Databases ....................................... 5
2.2.2 Images ......................................... 7
2.2.3 LSST-specificObjects ................................ 7
2.2.4 DataAccessServices................................. 8
2.2.5 User Catalog Data Support ............................. 8
2.2.6UserWorkspaceStorage .............................. 9
2.2.7 Data Access Permissions and Quotas ....................... 9
2.2.8 Support of Previous Releases ........................... 9
2.3 ComputingResources ................................... 10
2.3.1 Basic user compute services ............................ 10
2.3.2 Large-scale batch and parallel computing .................... 10
2.4 Authentication and Authorization ............................ 10
2.5 Resource Management ................................... 11
2.6 CybersecurityConsiderations ............................... 11
2.7 Additional Support Services ................................ 11
3 ThePortalAspect
11
3.1 GenericDataBrowsing ................................... 13
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
iii

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
3.2DocumentationDelivery .................................. 16
3.2.1 Context-DependentHelp .............................. 16
3.2.2 UserGuide ...................................... 17
3.2.3ReferenceandDeploymentManuals ....................... 18
3.3 CustomizedWorkflows ................................... 18
3.4 UseoftheLSSTStack .................................... 20
3.5 API Interaction with the Portal .............................. 21
3.6 Extensibility of Visualizations ............................... 21
3.7 Extensibility of the UI .................................... 21
3.8 Implementation ....................................... 22
3.9 Interface to the Alert Distribution and Filtering Services ............... 24
4 TheNotebookAspect
24
4.1 JupyterHub / JupyterLab service .............................. 24
4.2 Pre-installedLSSTsoftware ................................ 25
4.3 Pre-configured access to LSST data ............................ 27
5 TheAPI Aspect
27
5.1 Overview - VO services and custom LSST services ................... 27
5.2 Catalog and other tabular data access .......................... 29
5.2.1 TAPservice ...................................... 29
5.2.2 ADQLimplementation................................ 29
5.2.3 Return data formats ................................. 29
5.3 Imagedataaccess...................................... 29
5.3.1 Image finding ..................................... 30
5.3.2 Image etrieval .................................... 30
5.4 Metadata access ....................................... 30
5.4.1 DataReleases..................................... 30
5.4.2 Tables ......................................... 31
5.4.3 TableStructure .................................... 31
5.5 User Workspace access .................................. 31
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
iv

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
6The InterconnectednessoftheSciencePlatform
31
6.1 SharingData ......................................... 32
6.2 SharingQueries ....................................... 32
6.3 SharingState......................................... 32
7 ApplicationoftheSciencePlatforminsidetheProject
33
7.1 Data Management Teams ................................. 33
7.2 CommissioningTeam.................................... 33
7.3ScienceValidationTeam .................................. 33
7.4 ObservatoryOperations .................................. 34
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
v

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07

Back to top


SciencePlatformDesign

Back to top


1 Introduction
The“LSSTSciencePlatform”(LSP;LSE-319) eferstoasetofsoftwareandservicesthatis
used to provide data rights holders and project team members access to the LSST data and
supportforitsscientificanalysis,andisfrequentlyusedto eferinparticulartotheuser-facing
presentation of these components of the LSST system. The equirements for the Science
PlatformaredescribedinLDM-554.
The Science Platform provides access to both catalog and image data through a range of
cooperatinginterfaces(PythonAPIs,WebAPIs,andagraphicaluserinterface),aswellaspro-
viding dedicated computing and storage esources to users at the LSST Data Access Centers.
TheSciencePlatform’sprincipalcomponentsaredeliveredbytheLSSTconstructionproject
groups at SLAC (Database and Data Access), NCSA (middleware and infrastructure), LSST-
Tucson (SQuaRE), and IPAC (SUIT). These components make extensive use of the Science
PipelinessoftwaredevelopedbytheUW(AlertProduction)andPrinceton(DataReleasePro-
duction)groupstoprovidedataanalysissoftwareframeworksandalgorithms.
The Science Platform is visible to users through three “Aspects” giving access to a unified
system.
1.1 ThePortalAspect
ThePortalAspect,ahighlyevolvedversionofatraditionalastronomical-archiveGUI,provides
Web-basedqueryandvisualizationtoolsforalltheLSSTdataproducts,designedtofacilitate
the user’s discovery and query of the wide variety of catalog and image data products, illumi-
natetheconnectionsbetweenthem,enableexploratorydataanalysis,andprovidecontextual
access to relevant documentation. The Portal Aspect’s UI is extensible with additional com-
putation and visualization tools developed by users and collaborations. It provides access
notonlytotheProject’sreleaseddataproductsbuttostorageforfile-anddatabase-oriented
userdata,andfacilitatesthesharingofsuchdatabetweenusersandwithincollaborations.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
1

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
1.2 TheNotebookAspect
TheNotebookAspectprovidesaccesstoaPython-orientedcomputationalenvironmenthosted
at the LSST Data Access Centers through the “Jupyter Notebook” paradigm that has become
extremely influential in astronomy and in the wider scientific computing community in re-
cent years.
1
Through a Web-based notebook interface, users are able to run Python code
in close proximity to the LSST data archive, accessing and analyzing the data and generating
derived data products. Curated releases of the LSST software stack are made available to all
users of the Notebook Aspect, including all the framework and algorithmic software used in
theProject’sowndatareductionpipelines.Sufficientconfigurationandprovenancedataare
made available to permit users to eproduce the production processing steps and then to
customize them to their own ends or to assemble entirely new analyses. Batch computing
services and storage are available to users, with larger allocations equiring esource utiliza-
tion proposals, enabling them to perform analyses of a substantial scale. Users are enabled
toinstalladditionalsoftwareattheirowndiscretion,enablingthemtobringexistinganalysis
code to the LSST data, and take advantage of the wide variety of computational and visualiza-
tiontoolsavailableinthescientificPythonecosystem.
1.3 TheAPI Aspect
The API Aspect provides remote access to the LSST data, user data, and user computing re-
sources, through a set of Web APIs (many based on VO standards). The Web APIs will deliver
data in community-standard formats, including, e.g., VOTable, CSV, and FITS. The same Web
APIs are used internally in the Portal Aspect, and are also available in the Notebook Aspect.
NotebookAspectusers,runningtheirpersonalPythoncodewithintheDataAccessCenters,
willnormallyaccesstheLSSTdataand esourcesthroughtheDMPythonAPIs,whoseimple-
mentationwillgenerallybypasstheWebAPIs.
1.4 Architecture
Underlying all three Aspects are a common set of services hosting the LSST catalog and im-
agedataanduserdataproducts,auserstorage“workspace”,andprovidingaccesstoflexible
1
The LSST project is closely following current developments in the Jupyter project and expects to use the up-
coming “JupyterLab” evolution of the notebook concept. We are already using an alpha release of JupyterLab in
ourdevelopmentactivities.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
2

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
computing esources. Catalog data are stored in database systems providing highly scal-
able parallel access to the science data and Observatory metadata. The database systems
and the Web and Python data access APIs through which they are accessed support flexible,
user-defined ADQL queries. Image data is provided through a combination of image meta-
data search facilities and pixel data etrieval services that support cutouts and mosaicing.
Additional service provide access to a wide range of metadata describing the available data
products,theirstructureandtheirprovenance.
Further services support the creation of user image and catalog data products and facili-
tate the federation of user data products with the Project’s (e.g., permitting joins between
the Project’s catalogs and catalogs of additional, user-derived attributes). User data prod-
ucts created within the Data Access Centers are remotely accessible through the API Aspect.
The authentication and authorization services provided allow users to estrict access to the
datasets they create to specific other users and groups of users, supporting the work of sci-
entificcollaborations.
Byrelyingoncommonunderlyingservices,exchangeofdataandseamlessworkflowscross-
ing Aspects are enabled. For example, the data resulting from a graphically-driven query in
the Portal Aspect can be immediately accessed in the Notebook Aspect for further analysis.
Similarly,data etrievedorcreatedinthePythonenvironmentintheNotebookAspectcanbe
displayedoraccessedinthePortalaspect.
Both the Portal and Notebook Aspects of the Science Platform, as Web applications, are de-
signed to be usable by remote users without the installation of any software on users’ own
systems, with all computing and data access hosted at a Data Access Center. Nevertheless,
allthesoftware equiredtohosttheSciencePlatformispartoftheLSSTopen-sourcecode
base, installable on mainstream Unix-like platforms, and we anticipate its usability both by
non-Projectdataaccesscentersandbyindividualsandgroupsontheirownsystems.

Back to top


2 TheArchitectureoftheSciencePlatform
ThissectionpresentstheoverallarchitectureoftheLSSTSciencePlatform(LSP,nottobecon-
fused with the Science Pipelines stack" code), including its major components and interfaces
anddesignprinciples.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
3

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
2.1 DesignOverview
The LSP is a multi-tier architecture with distinct but inter-related user-facing Aspects" over
anunderlyinglayerofservicesthatinturnrestsoncomputationalinfrastructure.
2.1.1 Functional Architecture
The Portal Aspect, JupyterLab Aspect, and Web APIs Aspects provide three distinct user in-
terfaces. Internally, the LSP interfaces with the Data Backbone via the Data Butler client in-
terface and direct "native" Data Backbone interfaces to provide access to the Science Data
Archive and (for some instances) other data products. The LSP also has its own data storage
for Portal configurations, user notebooks, and other user files and databases. It interfaces
with compute provisioning services, authentication and authorization services, and esource
management, including a proposal management system to handle requests for esources.
Networking connects the LSP with internal systems and the external world and provides a
measureofcybersecurity.
The Portal Aspect is implemented using the language-agnostic Web APIs to etrieve data. The
JupyterLab Aspect can also use the Web APIs if the user desires, either through a VO client or
directly as a Web service.
2.1.2 Deployment Architecture
The LSP is designed to be deployable in multiple locations within the LSST DMS. The produc-
tion instances are the US Data Access Center, the Chilean Data Access Center, the Science
ValidationinstanceintheNCSAEnclave,andtheCommissioningClusterinstanceintheBase
Enclave.Additionalinstancesareusedforintegrationtestinginthe Integrationenvironment
andforsciencepipelinesdeveloperusageintheDeveloperenvironment.LSPdeveloperswill
ofcourseinstantiatecomponentsoftheLSPduringtheirowndevelopment.
The LSP is composed of multiple services that are containerized and deployed using Kuber-
netes. The underlying provisioning of compute esources is handled by the Provisioning and
DeploymentServicecomponentoftheLSSTDMS.Thetechnologytobeusedforprovisioning
in this component is being evaluated and may vary between instances. vSphere, HTCondor,
SLURM, and OpenStack are all potential alternatives. Provisioning is elastic: JupyterLab dy-
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
4

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
namically instantiates notebooks on demand, and the Portal and Web API services can be
expanded or contracted as needed. This elasticity and fault tolerance considerations equire
that the services hold minimal non-persistent state so that loss of a single service instance
hascorrespondinglyminimalimpactonusers.
2.2 DataAccessandStorage
SincethegoaloftheLSPisto etrieveandanalyzeLSSTdataproducts,dataaccessandthe
storage of user data are key aspects of the platform.
2.2.1 Databases
Since the LSST catalog data products epresent the best measurements of astrophysical phe-
nomena available to the project as well as metadata describing how data was taken and
processed, it is anticipated that most uses of the LSP will involve these catalogs which are
storedindatabases.
2.2.1.1 Databasecontentoverview
Severaltypesofdatabaseswillbeaccessibletothe
LSPinstances:
• Imageandvisitmetadata
• Catalogs
• Composite data (binary data in catalogs)
• Observatorymetadata(includingEFD)
•Referencecatalogs
Image and visit metadata includes information about the observation captured at the time.
It also includes information derived later from analyzing the data and metadata, such as
an accurate mapping from pixel coordinates in an image to sky coordinates or an estimate
of the photometric zeropoint in an image. The metadata will follow the Common Archive
Observation Model [6, 1] as its core. This metadata will be stored in relational databases and
madeavailableviaADQLqueries.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
5

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
Catalogs are described in detail in the Data Products Definition Document LSE-163. They in-
cludesummarymeasurementsofastrophysicalObjects,plusmeasurementsinimagesfrom
each epoch of observation and in difference images. These will also be stored in relational
databases and made available via ADQL queries. The catalog of all Alerts that have been is-
sued may be stored in a different database technology, such as a document database, and
may thus use a different query interface.
The above catalogs will include "BLOB" fields that provide large data items that are useful to
etrieve along with other catalog data but are not suitable for searching. Examples include
samplesoftheposteriorprobabilitysurfaceforamodel,photometricredshiftprobabilitydis-
tribution functions, or "heavy footprints" giving deblended pixel values for an Object. These
are generally stored in separate tables within the relational database as they are used less
frequently.
The Engineering and Facility Database, which contains a ecord of all commands, configura-
tion,andtelemetryfromtheObservatorysystems,willbetransformedtoprovidesimplified
query access to conditions during each observation. All data present in the original EFD will
be available in the transformed EFD with the exception of Internal, Sensitive, or Highly Sen-
sitiveinformation,asdefinedinthe InformationClassificationPolicyLPM-122,whichwillnot
be available to users other than project staff. The EFD does not contain Protected User data.
The T ansformed EFD will be stored in a relational database, accessible via ADQL queries.
ReferencecatalogsusedintheLSSTAlert,CalibrationProducts,andDataReleaseProductions
will be provided, also in relational databases. Additional catalogs, e.g. from precursor or
concurrent surveys in a variety of wavelengths, will also be provided as relational databases
depending on usefulness and available esources.
2.2.1.2 Databasetechnologies
Whilemanyoftheabovedatabaseswillbeimplemented
in conventional"relationaldatabasetechnology,thelargestcatalogswill equireadistributed
system to provide sufficient performance. The LSST-developed Qserv distributed database
willbeusedtoimplementthese. Thetwoimplementationsmayvarysomewhatintheir
support of ADQL features. Users creating their own catalogs (see section 2.2.5) will need
tospecifywhethertheyaretobedistributed,therebyselectingtheappropriateback-endim-
plementation.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
6

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
2.2.2 Images
LSST image data products include raw images, processed visit images, difference images,
coaddedimages, andtemplates. ThesearepartoftheScience ImageArchiveintheData
Backbone. All of these are accessible to the LSP.
Some data products are currently specified to be "virtual": they are egenerated on demand
fromotherpersistently-storeddataproducts,reflectingtheestimatesofthetechnology-dependent
space/time cost trade-off. Such virtual data products are egenerated for the LSP by services
thatarepartoftheLSPWebAPIs.
Images will be delivered in at least FITS format; other formats may be added. The LSST DM
internal image data format is currently FITS as well, but that may change in the future. Any
necessarytranslationbetweeninternalformatandexternalformatisperformedbyservices
thatarepartoftheLSPWebAPIs.
Imagedataproductstypicallycontainadditionalinformationbesidesimagepixels.Thisinfor-
mationincludesmetadatadescribingtheconditionsunderwhichtheimagewasgenerated,
its provenance and processing history, if any, and metadata derived from the image itself.
Much of this metadata can be expressed as simple data types in "headers" or similar mecha-
nisms,butsome,suchasmaskandvarianceplanes,pixel-to-skymappings,andpointspread
function (PSF) models, may be too complex for such a epresentation. When this information
is specific to the image, LSST DM generally stores this information in the same file as the im-
age pixels, as additional pixel planes or extension tables. The structure and content of these
will be documented for LSP users.
2.2.3 LSST-specificObjects
ThePythonobjectsusedbytheSciencePipelinescodedonotalwayscorrespondone-to-one
with persisted catalog or image data products in the Data Backbone. For example, an image
and its background model may be persisted in two distinct files. The PSF model associated
with the detections in an image may be stored in a file while the information about the de-
tections is stored in a database catalog.
The component files and catalogs are directly accessible via the LSP, but for convenience the
Data Butler client library provides mechanisms to define composites that combine multiple
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
7

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
componentsintoasingle etrievedPythonobject. SincethisusageisspecifictothePython
language, it is only directly visible to the user in the LSP in the JupyterLab Aspect.
2.2.4 DataAccessServices
TheLSPWebAPIsAspectprovidesservicesto etrievecatalogandimagedataproducts. Ac-
cess to catalog data products is provided by a Web service that supports the VO TAP protocol
[7]withqueriesspecifiedinADQL[13]. Accesstoimageandfiledataproductsisprovided
by a separate Web service that supports the VO SIA [8], SODA, and VOSpace [10] protocols.
The image and file service is also responsible for egenerating "virtual" data products and
converting from internal to external data formats as necessary. Both of these Web services
support asynchronous operations, as some requests will have long response latencies (e.g.
large-scale catalog queries or data egeneration). Access to metadata about images and files
is provided by TAP/ADQL queries on metadata tables, ather than a distinct metadata service.
More detail on these services is provided in section 5.
Within the JupyterLab Aspect, and internal to the Web APIs Aspect, the Data Butler client
interfacemaybeusedto etrievePythonobjects,anda"native"DataBackboneAPI maybe
used for high-performance etrieval of datasets. The Portal Aspect uses the Web APIs Aspect
exclusively for data access; it does not use the Data Butler nor the "native" Data Backbone
API.
2.2.5 UserCatalogDataSupport
Users may create their own catalogs within the LSP, a capability that is patterned after SDSS
SkyServer’s "MyDB". Catalogs may be derived from queries to the LSST catalogs or loaded
from computations or external data sources. As mentioned above in section 2.2.1.2, large
distributed catalogs meant for joining with the LSST data products must be identified so that
theycanbestoredappropriatelyintheQservdatabase.
AllthreeAspectswillprovidemechanismsforcreatingusercatalogs;thePortal’smechanism
will be implemented via the Web APIs, as usual. User catalogs may be accessed in the same
waysthatLSSTcatalogdataproductsareaccessed.
User catalogs are backed up for disaster recovery purposes, but they are not fully protected
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
8

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
againstusererror.
2.2.6 UserWorkspaceStorage
Users may create their own images and files within the LSP. These all live within a User
Workspace that is common across all three Aspects. All three Aspects will provide mecha-
nismsforcreatingand etrievinguserfiles. IntheJupyterLabAspect,thisisviaamounted
filesystem. In the Web APIs Aspect, this is via VOSpace or WebDAV.
User file workspaces are backed up for disaster recovery purposes, but they are not fully
protectedagainstusererror.
2.2.7 DataAccessPermissionsandQuotas
All LSST data products are available to all users with LSST data rights. Users who had LSST
data rights but have since lost them may be granted access to the data products available as
ofthetimeoflossbutnotnewerones.
Usersmaygrantotherusersorgroupsofusersaccesstocatalogsandfilesintheirworkspace.
Suchaccessmayberead-onlyor ead-write.
Per-usercatalogsandfilesarealimited esourcewithquotasdeterminedby esourceman-
agement (see section 2.5). Some "user" catalogs and files may be considered to be owned by
a group, with a distinct quota.
2.2.8 SupportofPreviousReleases
The LSST baseline currently anticipates that the data products contained in the last two Data
Releases (as well as Level 1 Alert Production data products) will be available through the LSST
SciencePlatform. DataproductsfromotherDataReleaseswouldneedtobe etrievedviathe
BulkDistributionservice;theywouldnotbevisiblewithintheLSP.
A request to have the catalogs from all Data Releases be available through the LSP is being
evaluated.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
9

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
2.3 ComputingResources
TheLSPprovidesaccesstocomputing esourcestoenableuserstoanalyzeLSSTdataprod-
ucts,aloneandincombinationwithuser-provideddata.
2.3.1 Basicusercomputeservices
UsersofthePortalAspectsharecomputing esourcesallocatedtothePortalasawhole. The
samestrategyisusedfortheWebAPIs. Withinthoseshared esources, per-useraccount-
ingisperformedfor esourcessuchasqueriesandtemporarydiskspacesothat esource
management policies may be implemented if equired.
Each user of the JupyterLab Aspect is assigned a virtual machine per notebook. The virtual
machine can be selected from a menu of preconfigured installations of LSST and other soft-
ware, or users can customize a virtual machine and save it for future use. The esources
allocatedtothevirtualmachinearedefinedbypolicy.
2.3.2 Large-scale batch and parallel computing
Users of the LSP (any Aspect) may trigger large-scale, long-latency, asynchronous data pro-
cessing.ThisprocessingishandledbysubmittingjobstoabatchsystembasedonHTCondor.
MechanismsareprovidedinthethreeAspectstomonitorthestatusofbatchjobsand etrieve
their results. Instances of the workload and workflow management tools and services used
by the LSST Data Facility to manage large-scale productions will be made available to each
LSP instance, although these will be separate and distinct from the production instances, of
course.
2.4 Authentication and Authorization
IdentitymanagementandassociatedrolesandauthorizationsareprovidedbyaunifiedLSST
system as described in LSE-279. This system allows the use of external identity providers
such as the Chilean COFRe federation [2] or the US InCommon [3] or GitHub; those verified
external identities are then mapped to internal LSST identities. All uses of all instances of the
LSP,includingtheWebAPIsAspect,areauthenticated.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
10

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
2.5 ResourceManagement
LSPusersintheDACswillreceiveacertainquotaof esources(compute, storage, query,
bandwidth, etc.). Usage beyond that quota will be allocated by a committee. Outside the LSP,
a proposal management system will track requests and allocations that will then be imple-
mented by per-user esource management features within each LSP component.
2.6 Cybersecurity Considerations
The Data Access Center instances of the LSP are the only ones exposed directly to the pub-
lic Internet. The Science Validation, Commissioning Cluster, Integration, and Developer in-
stancesareonlyreachablefromLSST-internalnetworks,helpingtoreducetheattacksurface.
The DAC instances are isolated (on a different LAN) from the production Level 1 and Level 2
domains within the NCSA enclave. Although they may share underlying computing esources,
such esourcesareprovisionedforeitherproductionorDACbutnotbothatthesametime.
The LSST data products within the Data Backbone will be read-only to the DAC LSP instances.
The Data Backbone interfaces are all authenticated, but they will be used for some types
of user data, so the Backbone must be hardened against attack (or accident) from the LSP.
Provisioning separate virtual machines per user in the JupyterLab Aspect helps to prevent
unwanted sharing of information between users. The Portal Aspect and Web APIs Aspect
need to be hardened in the same ways as any Web-enabled service, including sanitizing user
inputs,avoidingcross-sitescripting,etc.
2.7 AdditionalSupportServices
System management and monitoring services for the LSP will be provided by the infrastruc-
ture layer supporting each instance. Monitoring at the system and application level will be
exposedtoLSPuserswhereconsistentwithuserprivacyandsystemsecurity.

Back to top


3 The Portal Aspect
The Portal Aspect of the LSST Science Platform provides a Web user interface for the sci-
ence community to discover, access, explore, analyze, and download the LSST data. It pro-
videsqueryandvisualizationservicesforalltheLSSTdataproducts,designedtofacilitatethe
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
11

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
user’s discovery and exploration of the wide variety of catalogs and image types available. It
illuminatestheconnectionsamongthedataproducts,includingthedataflowsthatproduced
them,andprovidescontextualaccesstorelevantdocumentation.
ThePortalAspectprovidesforbothsimpleform-basedqueriesandtheconstructionofcom-
plex queries in ADQL. It provides a sophisticated set of data visualization capabilities for in-
specting the results of queries, both for catalog and image data, and performing basic ex-
ploratorydataanalysisontheresults.
It provides access not only to the Project’s released data products but to storage for file-
anddatabase-orienteduserdata,andfacilitatesthesharingofsuchdatabetweenusersand
withincollaborations.
TheimplementationofthePortalAspectwithintheoverallSciencePlatform,anditsreliance
on common underlying services with the other Aspects, allows users to identify and query
data in the Portal and then access the results in the Notebook Aspect, where they can per-
form custom analyses and draw on the full variety of computational and visualization tools
available in the Python language. Conversely, users can also query or create data in the Note-
bookAspect,constructingcomplexADQLqueriesandprocessingworkflows,andthenmake
theresultsavailableinthePortalforvisualization.
The Portal Aspect is being designed to serve users with a wide range of levels of expertise
and knowledge of the LSST survey and data products. It provides new and casual users with a
guided introduction to, and discovery of, all the LSST data products, but also provides expert
userswithasourceofquick eferenceinformationontheavailabledataproductsandtheir
structure,andtheassociateddocumentation.
ThePortalAspectisintendedforuserswithdatarights. Itwill,therefore,startwithalo-
ginscreenatwhichusermayusethecredentialstheyhave egisteredwiththeLSSTproject
to identify themselves. All interaction with the detailed survey data will equire their user
accounttoholdtheappropriaterights.
2
Howmuchbroadsummaryinformationmaybepro-
vided to unauthenticated users through the Portal itself — e.g., documentation on the LSST
project,asummaryoftheprogressandcoverageofthesurvey—versusthroughotherpublic
faces of the Project is yet to be determined.
2
Note that although the
broadcast
of alerts via VOEvent (or a successor) is world-public, the use of the
Science
Platform
to browse or query the historical alert database is limited to data rights holders.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
12

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
Since all users will be authenticated, the Portal will be able to provide identity-based persis-
tent state that allows users to start their work in one browser and then continue it elsewhere.
How much user state will be identity-based and how much will be session-based is still being
considered and will be adjusted in response to user feedback.
The Portal Aspect also provides convenient access to external data relevant to the under-
standing and exploitation of the LSST data products: to eference catalogs maintained on
LSST systems, to a selection of specific publicly available data archives, and to any data
archivesavailablethroughstandardVOservices.
3.1 GenericDataBrowsing
Akey equirementonthePortalisthatitprovidesfordiscoveryofalltheLSSTcatalogand
image data products as well as the Observatory metadata such as the contents of the En-
gineering and Facilities Database. By relying on the creation of extensive metadata on the
structure of the data products in the Science Pipelines code and other data sources, and on
its delivery via the deployment of the data for release through the API Aspect and the under-
lyingdatabasesandservices,thePortalisabletoprovidecontent-sensitivedocumentation,
query,exploration,andvisualizationofeveryreleasedtable.
Driven by the metadata available through the API aspect, the Portal will be able to display to
the user the available data releases, and all the tables available in each release and in Level 1.
The Portal will allow the user to select the the data product to search on, provide a search
formfortheselecteddataproduct,generatethesearchrequestbasedonuser’sinput,submit
thesearchanddisplaytheresults. Ifsufficientmetadataareavailable,adetailedsearchform
will be constructed that allows the user to specify constraints on any column without having
to know how to write an ADQL query. The search form will also always provide a means for
the user to specify an ADQL query directly. In either case, the Portal will then pass the ADQL
alongtotheAPI Aspecttodothesearch.
Given appropriate metadata (e.g., IVOA UCDs), the Portal will recognize specific columns as
epresentingspatialdata,particularlyskycoordinates,andwillprovidecontextuallyappropri-
atesearchformssupportingcommonly-usedqueriessuchasconeandboxsearches.
If the selected data product is catalog or other tabular data, the search results will be dis-
playedasatableinthePortal. Thetableviewerwillprovideafamiliarsetofbehaviorsfor
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
13

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
manipulating the results, such as sorting by column, setting the page size, and customizing
the column selection. In addition, it will provide basic data analysis features such as the abil-
ity to filter on a column or add a synthetic (calculated) column. For any row in a query result,
the system will provide the ability to obtain a metadata-driven formatted view of all the data
in the row — a “property sheet” for the row — which can be extended to the full width of
the underlying table(s) (in effect performing a “SELECT *” on demand for a selected row in a
narrowerqueryresult).ByprovidingadequatemetadatathroughouttheLSSTdataproducts,
the Portal will be enabled to, for instance, display values together with their uncertainties in
thesepropertysheets.
ThePortalwillrecognizecolumnsthatarekeystoinformationinothertablesandwillfacilitate
navigationtothosetablesfortherelatedinformation.
The Portal will provide for straightforward plotting of tabular data. It will provide for the
creation of scatter (or line) charts for any selection of X and Y axes from the available columns
or arithmetic expressions based on combinations of columns, with assistance from the UI in
actions such as finding columns of interest to use or verifying the spelling of their names.
The Portal will also have the ability to create histograms on demand for columnar data or
expressionsoncolumnardata.
If the selected data product is image data (i.e., if the search was on an image metadata table),
the Portal will be able to use the associated metadata to construct queries for the images
themselvesanddisplaythemtotheuser. Avarietyoffunctionswillbeavailabletoadjust
and interact with the image displays: stretch, color change, zoom, rotate, flip, crop, measure
distancebetweentwopoints,overlayavarietyofcoordinategrids,addmarkers,uploadDS9-
style “region files”, plot the telescope field of view, etc.
Tables of catalog data containing IDs linking to the images on which the measurements were
made can be used in the same way to drive browsing over image displays.
The user experience when viewing image data will depend on the latency of availability of
the image data from the underlying services. Currently only coadded images and a 30-day
cache of calibrated single-visit images (and cutouts from such images) are expected to be
available with low latency. The Portal will be able to deliver views such as “flip books” of
cutouts from the single-epoch images for a given region of sky for the entire survey, but only
upon submitting a request for the data, which may take of the order of several hours to
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
14

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
fulfill. The user workspace may be used to save the results of a limited number of such data
requests,basedonuserquotas,forlaterre-use.
The table, charting, and image display components of the Portal will be linked through an
underlying shared data model. As a result, the Portal can provide “brushing and linking”
and similar capabilities, reflecting selections and operations on data in one component into
theothercomponents. Forinstance, theresultsofanObjecttablecatalogquerycanbe
displayedsimultaneouslyasatable,anoverlayondeepcoaddimages,andacolor-magnitude
diagram. Selection of a region in the color-magnitude diagram will be immediately reflected
as a highlighted selection of rows in the corresponding table and of object markers in the
imageoverlay.Similarly,acolumn-basedselectioninthetablewillbereflectedinthediagram
and on the image.
All query results — tables and images — will be available for download in either their original
orinteractivelymodifiedforms.Datamaybedownloadedeithertotheuser’sremotesystem
(i.e.,thesystemrunningthebrowser)ortotheuser’sSciencePlatformworkspace.ThePortal
Aspect will also support the creation of user databases based on tabular query results.
Inaddition,allqueriesonLSSTorLSST-hostedperformedbythePortalthroughtheAPI Aspect
willbeassociatedwiththeuserandwillbereadily etrievablethroughtheNotebookAspector
directlythroughtheAPI Aspect.Thiscapability,furtherdiscussedbelow,isthekeytoenabling
seamlessscientificworkflowsthatcrosstheAspects. Itwillalsobepossibleforauserto
browse their query history and select individual queries for re-execution or for download of
the previous results (if they are still in the results cache).
The Portal Aspect will provide for access to user data, both in the Science Platform’s user
workspace (which accommodates file-oriented data) and in user-created databases. It will
provide a traditional hierarchy-browser capability for the workspace, with the ability to in-
spect and visualize the contents of image or data table files of recognized types (e.g., FITS,
VOTable,CSV).Theworkspacebrowserwillalsoprovidebasicmanagementfunctionsforthe
organizationoftheworkspacehierarchy,renaminganddeletionoffiles,andthelike.ThePor-
talAspectwillprovidefordiscovery,query,andvisualizationofuserdatabasesusingthesame
generic data browsing tools as for the LSST data, though the sophistication of this interface
willdependontheavailabilityofadequatemetadatafortheuserdatabases.
ThegenericdatabrowsingcapabilitiesofthePortalwillalsobeavailableforselectednon-LSST
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
15

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
eference catalogs that will be hosted at the LSST Data Access Centers, for purposes of cali-
bration, data quality analysis or user convenience. The semantic knowledge of the LSST data
that the Portal is planned to exploit may be more limited for these hosted external datasets,
butallthebasicfunctionsofthePortalwillstillbeavailable. Inaddition,thePortalwillbe
able to access any publicly available dataset that has a VO-standard interface, particularly a
TAP/ADQLinterface.Semanticfeaturesoftheinterfacewillbeenhancedwhenexternaldata
(hosted or remotely accessed) are organized according to the CAOM model (in the case of
observational metadata) and when queries eturn well-curated UCDs. Query results will be
capable of being saved to the user workspace.
3.2 DocumentationDelivery
A key function of the Portal is to connect the user to documentation on the Portal itself,
theSciencePlatformmoregenerally,thedatastructureandquality,thedataprocessing,the
survey, and the LSST itself. Much of this documentation will be created by other parts of the
project,butthePortalwillprovideastartingpointfordiscoveringrelevantdocuments.
3.2.1 Context-Dependent Help
ThePortalwillprovideonlinedocumentationinavarietyofways:implicitlythroughthenam-
ing and labeling of UI elements and screens, via “tool tips”, and via dedicated “Help” buttons
oricons.
UI elements such as the label and title for a window, an input form, an input field, a column
in a result display, a function name, etc. are the first points of entry for users to understand
and interact with the LSST data. Such labels and titles have to be well thought out, as precise
and unambiguous as possible, but also human-readable. This may mean that many entities
mayneedtohavebothan“internal”name, suchasthetruenameofatableinadatabase,
anda“commonname”thatfacilitateshumanunderstandingandspeechabouttheentity. In
such cases, if the common name is displayed by default in the UI, the UI must allow the user
to discover
and copy andpaste
the true nameso that it can be usedin contexts, such as typing
ADQLqueries,whereitis equired.
Duetothelimitationsonscreenrealestateandresolution,andthedesiretodesign“efficient”
layouts that devote as much space to the data as possible, labels and titles cannot generally
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
16

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
containalltheusefulinformationavailable.Asecondlevelofcontext-sensitivedocumenta-
tion is, therefore, provided in the form of “tool tips” or hover-over text. The design will allow
for descriptive text to be associated with all data product entities, notably table and column
names, and displayed on mouse-over. (The actual provision of this documentation will be a
collaborativeeffortacrossDataManagement.) Inaddition,theavailabilityofmetadatasuch
as units and IVOA UCDs will permit the delivery of context-dependent help in the interpreta-
tion of displayed values and in the use of search form fields.
Thefinalcategoryofcontext-dependentonlinedocumentationwillbeassociatedwithexplicit
“Help” icons in various places in the interface. These will take the user to detailed documen-
tationinanotherwindow,withastrongpreferencefor“deep-linking”thattakestheuserfirst
tothemostrelevantpointinadocument. Itisbythismeansthatweexpecttoprovideac-
cesstothelargerworldofdocumentationonthedataprocessing, dataquality eports, and
documentationonthesurveyandtheObservatoryitself.
3.2.2 User Guide
While the aim of the Portal design will be to make its use, and the features it provides, as
evident as possible through the UI elements themselves, we nevertheless expect to provide
a user guide to the Portal. The user guide will describe the major capabilities of the Portal,
the options available associated with each major screen and function, and design specifics
suchastheuseofuserstatevs. sessionstate. Itwillgrow,withuserfeedback,tocontain
additional information of a “FAQ” nature that has been found to be useful. The user guide will
contain examples that alert users to the breadth of capabilities they can expect to find in the
Portal Aspect and motivate their use with scientific examples. Elements of the user guide will
be linked to appropriate places in the UI as context-dependent help, but will also be available
as part of the full document.
Some parts of the user guide will naturally cross Aspects of the Science Platform — for in-
stance,auser-friendlyexplanationofthe“userworkspace”willnaturallycoveritsavailability
and use from all three Aspects.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
17

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
3.2.3 ReferenceandDeploymentManuals
Along with the user guide, the Portal development project will also produce eference and
deployment manuals. The eference manual will include API documentation for both the
client and server components of the Portal, as well as documentation on the interfaces the
Portal relies on with the rest of the LSST system.
The deployment and maintenance manual will provide the information equired to maintain
theoperationofthePortalinthecontextoftheLSSTinfrastructure,includinginformationon
how to run and configure its components, and how to ensure that they are able to commu-
nicate successfully with the rest of the Science Platform components and infrastructure. It
is intended primarily to be useful to the project’s operations staff, but it will also be aimed
at providing information useful to those who may wish to set up additional instances of the
SciencePlatformoutsidetheproject.
3.3 CustomizedWorkflows
Inadditiontothemetadata-drivengenericdataexplorationcapability,thePortalAspectwill
provide dedicated workflows aimed at improving the user experience for access to a small
numberofthemostcommonlyaccessedLSSTdataproducts.
This involves some extra effort to design the specific screens, UI actions, and connections
from one to another. It involves a tradeoff between the improvement in user experience
and a potentially increased exposure to the need to modify these elements of the Portal to
account for changes in the structure of the data, e.g., from one data release to the next. It can
also be seen as a tradeoff between providing certain functionality directly versus extending
thecapabilitiesandgeneralityofthemetadatasystemtounjustifiablelevels.
ExamplesofPortalfeaturesinthiscategorymayinclude:
• In caseswhere thescientific informationof theinformation inone table equires a
more complex query involving, for example, calibration information from another ta-
ble(e.g.,thesubtractionofphotometriczero-points),thePortalwillprovideUI support
for the single-click application of that calibration to queries generated from the stan-
dard screens. It will then also provide contextual help explaining what has been done,
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
18

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
sothattheusercanlearntheunderlyingquerylanguagethatis equiredandthenbe
able to perform similar queries, e.g., in the Notebook aspect, without the UI support.
• UI guidancefortheinterpretationofparent-childrelationshipsbetweenentitiesrelated
bydeblendingalgorithms.
• Property sheet screens for the most important tables (e.g., Visit, Object) that organize
and present the data in more user-friendly ways that depend on expert knowledge of,
e.g., what the most scientifically interesting and most commonly used attributes are,
or which attributes may be the most likely to be of interest to see close to each other
on the screen, or that provide explanatory text or visual guidance beyond what can be
readily epresented by the metadata and general contextual help system.
• Customized UI actions available in property sheets or the table viewer for key tables,
providingagreaterrichnessand/oruser-friendlinessoffollow-onsearches.Forexam-
ple, we expect to provide a simple “Display light curve” UI action on the property sheet
foranObjectoraDIAObject. Incontrast, inageneric-browsingversionoftheObject
property sheet, the same functionality might be met by a “Search related tables” UI el-
ement that would let the user pick the “ForcedSource” table from a list of tables with
foreign-keycolumnspointingbacktoObject,andthenpicktheappropriatefluxcolumn
from the ForcedSource table. In cases where such simplified functionality is provided,
the contextual help system will be used to ensure that the user understands what the
actual search performed is, including the ability to review the ADQL used, so that the
user can then decide to customize the search differently from the default.
• Prioritization of columns in tabular data displays for the most important tables, making
— in the absence of user requests to the contrary — a consciously designed subset of
the columns in these tables more easily accessed in the table viewer, in a column picker,
etc.
• Custom designs of mouse-over inspectors for data points in plots generated from key
tables.
• Pre-configured, UI-selectable actions for generating commonly used displays or calcu-
latedcolumns.Forexample,forphotometricObjectdatawheretheactualdatabaseta-
blecontainsper-bandmagnitudes,theUI mayprovidesingle-clickactionsforchoosing
todisplaycolors,orevenrequeststandarddisplayslikecolor-colororcolor-magnitude
plots.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
19

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
• Different default settings for parameters such as image stretch for different types of
images(e.g.,forcoadds,differenceimages,andcalibrationflats).
The successive versions of the Prototype Data Access Center deployments of the Portal, and
the use of the Portal during commissioning, will be used to gather experience and user rec-
ommendations to guide the development of the most useful such customizations as the op-
erationseraapproaches.
ThePortalwillbedesignedtofacilitatethecontinueddevelopmentofadditionalcustomiza-
tionsduringtheoperationsera,asuserexperienceaccumulates.
3.4 UseoftheLSSTStack
The basic functionality of the Portal Aspect is based on the external persisted forms of the
LSST data provided through the API Aspect’s Web interfaces, which eturn “on the wire” data
in community standard forms such as FITS image and table files and VOTable. In many cases
the Portal implementation is able to work directly with these persisted forms, without de-
pendencies on the LSST pipeline code base in order to interpret them. However, among the
LSST data products there are also ones for which there are no semantically meaningful com-
munity standard forms, and for the interpretation of which the use of the LSST code is very
useful, if not indispensable.
3
Likely examples of such data objects are source footprints and
“heavyfootprints”(bitmapsandpixelfluxdata,respectively,forsinglesourcesandgroupsof
sources),andimagePSFmodels.
Inordertooffervisualizationsofthesedataproducts,thePortalAspectwillinvokeLSSTstack
code to transform them into useful visual forms such as image overlays, plots of PSF varia-
tion, and the like. The Portal Aspect will only attempt to provide a limited set of such standard
visualizations,recognizingthattheNotebook’sPythonenvironmentandtherangeofvisual-
ization toolkits that will be available there will enable the user community to develop a rich
setofspecializedtoolsforthesepurposes.
Note that even in cases where there are LSST data products for which no dedicated visual-
3
“Not indispensable” because ultimately all the persisted data object forms equire complete documentation,
of a quality and depth which would permit the independent development of code to interpret them. However, in
somecases,duetothecomplexityofthedatamodel,weanticipatethatsuchareimplementationwouldbecostly
todevelop.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
20

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
ization is provided, the Portal Aspect will still provide the ability to inspect the data object
contents. For example, for data products epresented by complex FITS table models, the
tables themselves will be viewable even if the Portal does not directly provide a semantic
interpretation of the data product they epresent.
Theextenttowhichthiswillbe equiredisstillbeingdetermined,becausethefullpersisted
datamodelfortheLSSTdataproductshasnotyetbeenspecified. Asabove, development
will prioritize the development of visualizations for the most commonly used or otherwise
importantdataproducts.
3.5 APIInteractionwiththePortal
ThePortalAspectwillprovideAPIsforinteractingwithitscapabilitiesandwithusersessions.
It will be possible for a user to programmatically upload image and tabular data and open a
visualizationinacurrentPortalsession—again,withthesophisticationoftheresultsdepen-
dentontheprovisionofmetadata. Itwillalsobepossibletoprogrammaticallyquerythestate
of a session, in support of users’ ability to begin a workflow in the Portal and then continue it
programmaticallyoutsidethePortalAspect – mostnotably,insidetheNotebookAspect.
3.6 Extensibility of Visualizations
The Portal software infrastructure is being developed to permit users to create visualiza-
tions not in the basic Portal software release. The Portal relies on the open-source scientific
graphics library Plotly for the generation of a variety of types of plots. Plotly provides well-
documented APIs and data structures for the creation of a wide variety of scientific and other
graphics. While only a few of these will be used in the standard Portal configuration, the Por-
tal will be designed to permit the use of the underlying interfaces to add custom plots under
usercontrol.
3.7 ExtensibilityoftheUI
The Portal Aspect’s UI will be extensible with custom actions associated with data types and
selections.Forexample,inthePrototypeDataAccessCenterthecapabilityhasalreadybeen
demonstrated of creating a button in the image-viewing UI that allows the connection of a
selected region on an image to user code that performs an action based on that selection.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
21

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
3.8 Implementation
The planned implementation of the Portal Aspect is based on an evolution of Firefly, origi-
nallydevelopedatCaltech/IPACfortheNASASpitzerHeritageArchiveandthe IRSAarchives.
Firefly is a framework and a set of libraries for the development of Web-based client-server
applications for accessing astronomical archives. It comes with a native set of capabilities:
basicdata-searchfunctions,astronomicalimagevisualization,andsophisticatedtabulardata
visualizationcapabilitiesincludingashareddatamodelthatpermitsthedeliveryofbrushing-
and-linkingfunctionalityacrosstabular,chart,andimagedisplays.Fireflyalsoprovidesaset
of JavaScript, Web, and Python APIs, allowing the construction of customized Web applica-
tions,butalsoallowinguserstoaccessthevisualizationcomponentsintheirownsimpleweb
pages(orJupyternotebooks)withouthavingtobuildacompletewebapplication.
ManyFirefly-basedapplicationsareinoperationsin IRSA-operatedarchives,suchastheSpitzer
HeritageArchive,theWISE ImageService,thePlanckUSDataCentervisualizationservice,PTF
ImageService,andthenew IRSAFinderChartapplication.
Conceptually, Firefly-based applications are organized into three tiers: Client, Server, and
DataServices. TheClienttierisresponsiblefor enderingthedisplayofthedataandprovid-
ing the UI. The Client is a native Javascript application based on the React framework. The
Javascript APIs of the components of the application allow for the straightforward construc-
tion of a wide variety of portal UIs combining images, tables, and plots of the tabular data.
StandardScientificvisualizationsaregeneratedintheClientviathePlotlylibraries,whileas-
tronomical image visualizations are handled in a Google Maps-like way with tiled endering
performedontheServer.
The Server tier manages searches, the Firefly shared data model that enables data linkages
acrossvisualizations,andprovidestheinterfacetoothersupportingservicessuchasauthen-
tication providers. The Server tier provides an abstraction layer for creating “search proces-
sors”usablefromtheClient.Theseareplugininterfacestoavarietyofdatasources,providing
support both for executing queries via standard external protocols such as IVOA TAP and for
working with data services with custom protocols. The Server handles search requests, dis-
patchesthemtotheproperdataservice,obtainstheresults,andpreparesthemfor endering
anddisplay.TheServeralsoprovidesexploratorydataanalysisfeaturesthatcanbeapplied
to the data it has received.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
22

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
ThecoreoftheServerisaJavaapplication, hostedonaTomcatserver. Communication
betweentheClientandServertiersisthroughacombinationofstandardHTTPSrequestsand
a WebSocket interface. The Server tier provides the Web APIs — and wrapper Python APIs for
them — that enable driving and querying the server state from other system components or
byusers.
TheServertierincludestheabilitytoextendthecapabilitiesofFireflythroughmicroservices.
This facility will be used in the Portal Aspect to provide visualizations for LSST-specific “blob”
data products for which the DM stack code will be used to convert the data product into an
artifactsuitablefordisplay.
Finally, the Data Services tier actually performs searches and other data etrievals, and is
generally composed of components provided by the underlying data archives. In the case of
the Portal Aspect’s Firefly application, the Data Services tier essentially comprises the data
accessservicesprovidedbytheAPI Aspect.
As part of the LSST project, Firefly will be extended with more sophisticated metadata mod-
els enabling the the implementation of the advanced generic-browsing behavior described
above, support for all-sky visualizations based on HEALPix and the HiPS file format, and with
the ability for users, via the Firefly APIs, to call on advanced visualizations provided by the
Plotlylibrary,beyondthoseusedforFirefly’snativefunctions.
In addition, in developing the Portal, the SUIT team is reviewing the existing heritage of UI
elementsprovidedbyFireflyandrevisingtheappearanceandworkflowstotakeadvantageof
contemporarydevelopmentsinuserexperiencedesign,andtoprovideauniformexperience
acrossthetool.
Keydesignguidelinesinthisprocessare:
• theefficientuseofdisplayspace,withasmanypixelsaspossibledevotedto endering
data in preference to static UI elements;
• in data-viewing screens, a corresponding preference for more context-sensitive “pop-
up”UI elementsversusstaticbuttons;
• consistencyofpresentationofcommonentities(e.g.,skycoordinates)acrossthePortal,
withanemphasisontheeaseofselectinganentityinoneplaceintheUI andusingit
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
23

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
(e.g., by copy and paste) in another without editing;
• consistency of layout and of the behavior of UI elements, including issues like the use
of modal versus modeless dialogs and inspectors, and the positioning and meaning of
controlssuchas“OK”,“Apply”,“Cancel”,and“Dismiss”.
3.9 InterfacetotheAlertDistributionandFilteringServices
Beyondprovidingthearchive-accessfunctionalitydescribedabove,thePortalAspecthasthe
additionalroleofprovidingdata-rightsholdersasimpleUI foraccesstothealertdistribution
and filtering facilities provided by the Project. This is essentially a “mini-broker”; unlike the
envisioned community alert brokers, the LSST-specific facilties will have access only to the
information in the LSST alert messages themselves. The UI will allow users to establish and
control subscriptions to the alert stream, and to select from a modest set of pre-defined
filters, or submit simple user-defined filters, to be applied to the alert stream they receive.
The Portal UI for this will be closely tied with screens for the querying of the alert database
and the visualization of its contents, to provide convenient workflows for users interested
in the time-domain LSST data. We anticipate being able to provide data access workflows
that enable users to tie the information actually present in alerts to additional data, though
context-sensitivesearchesandthelike.

Back to top


4 TheNotebookAspect
The Science Platform offers in-house scientists and engineers as well as (in operations) gen-
eral science users the opportunity to interactively explore the LSST dataset from a python
notebookenvironment.
4.1 JupyterHub/JupyterLabservice
ThenotebookserviceisascalablekubernetesdeploymentwithJupyterHubspawningindivid-
ual JupyterLab pods for each user. The basic architecture is outlined in Figure 1. JupyterLab
is a rich notebook architecture that offers terminal access and dashboarding in addition to
the traditional Jupyter Notebook concept and the use of JupyterHub allows it to be scaled to
demandasinfrastructureallows.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
24

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
A user of the JupyterLab sevice will hence be able to interact with LSST services in the same
waytheymighthavewithpythonscriptsinashellenvironment. Theycanuseourpython
interfacestothewebAPIsandtheButlerfordataaccess,theinterfacetotheworkflowsystem
forbatchprocessing,visualisationviabokeh,Firefly,matplotlibetc.
Thecapabilitiesoftheservicearetiedtomajorprojectmilestonesoutlinedinthetablebelow
in a series of phased releases. Each release is primarily targeted at a different audience - see
Section 7 for a summary of how the capabilities of the Science Platform as a whole are going
to be used by teams inside the project.
Version Audience
TargetL2Milestone Shimsok?
proto SQuaREandArch
none
y
alpha DMdevs,Commissioningteamtraining LDM-503-1
y
beta CommissioningTeam
LDM-503-9
y
1.0 Commissioning(ComCam)Capabilities LDM-503-12
n
2.0 ScienceVerificationCapabilities
LDM-503-14
n
3.0 GeneralScienceUserCapabilities
LDM-503-16
n
Noteinparticularthatinadditiontotheoriginallyplanned“Level3”scienceusercapabilities,
the JL service is the primary interface of the LSST System Scientist to Commissioning. This
introducesscopenot equiredbythegeneralscienceuser,eg. timelyaccesstodatainthe
Engineering Facilities Database and the ability to rapidly deploy instances of the service on
theCommissioningCluster.
For analyses equiring significant data processing, a process for transitioning a trial-and-error
notebook into a batch job involving execution over a large dataset will be provided.
FordetailedarchitecturaldiscussionsoftheJupyterLabdevelopmentwork-in-progress,its
interfaces to other components and the current schedule for major and internal milestones
seeSQR-018.Astechnicalchoicesarebaselinedtheywilltransitiontothisdocument.
4.2 Pre-installedLSSTsoftware
An important advantage of the JupyterLab environment is that it is already set up and opti-
mised for running a variety of LSST data. We achieve this by building containers directly from
the weekly releases created by our continuous integration system and then deploying them
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
25

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
Github
OAuth2
User1
Browser
jupyterlabdemo.lsst.codes
nginx proxy pod
(TLS termination +
reverse proxy to JupyterHub)
In green: SQuaRE deployment on GKE
SQuaRE JupyterHub pod
JupyterHub
SQuaRE
Authenticator
JupyterHub
UserDB
Kubespawner
JupyterLab
user1 Pod
JupyterHub
User Routing Proxy
auth
User2
Browser
https
JupyterLab
user2 Pod
auth
Work in Progress!
Does not represent architecture
of Science Platform
NFS
Storage Pod
FIGURE 1:Architectureoftheproof-of-concept(alpha)JupyterLabdeployment
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
26

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
to JupyterHub which asks the user which version to use to spawn their pod. This means the
users always have a choice of cadence (latest weekly or last stable) in their environment de-
pending on their risk tolerance versus appetite for the latest feature. We plan to make other
complementarythird-partyastronomicalpackages(eg. sextractor)availabletoo.
4.3 Pre-configuredaccesstoLSSTdata
TheJupyterLabservicewillmakeuseofstandardLSSTstackfeaturessuchasDAXservices,the
Butler,andtheirinitialisationthroughtheSuperTaskActivatortoallowpython-levelaccessto
LSST data. After that users can use a mix of their own code and stack classes to manipulate
and visualise the data. An example notebook running on our prototype deployment using
classes from the LSST stack is shown in Figure 2.

Back to top


5 TheAPI Aspect
TheWebAPIsprovidelanguage-agnostic,location-agnosticauthenticatedaccesstoLSSTdata.
Theyareintendedtobeusedbyclienttoolsandautomatedprocesses. Theyarenotopti-
mized or supported for direct human use via a browser, although they can be used that way.
TheavailableendpointsandtheURLstructurestheyacceptwillbedocumented.
The Web APIs are undergoing detailed design. Initial prototypes exist that provide non-VO-
compliant access to catalogs, images, image mosaics, and image cutouts. This section de-
scribes the ultimate goals for the LSP Web APIs and documents design features where they
arealreadyknown.
5.1 Overview-VOservicesandcustomLSSTservices
The Web APIs support standard VO protocols where possible. They will also support exten-
sions to the VO protocols and additional custom LSST services that are useful for the Portal
Aspect or JupyterLab Aspect. To the extent that it makes sense, such extensions will be pro-
posedtothe IVOAforincorporationintothestandardprotocols.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
27

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
FIGURE 2:Exampleofstar-galaxyseparationanalysiswiththeJupyterLabprototype
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
28

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
5.2 Catalogandothertabulardataaccess
Catalogs, metadata, provenance, and other tabular data are accessed through the dbserv
endpoint.Formetadataandprovenance,tableschemas,whenpossible,willbereusedfrom
VO or community standards, such as CAOM2 [1] and others.
5.2.1 TAPservice
dbserv is the LSST implementation of IVOA’s TAP interface. dbserv will target compliance with
TAP 1.1 once it is recommended by the IVOA.
5.2.2 ADQLimplementation
Internal to dbserv is an ADQL parser. This parser will target compliance with ADQL 2.1 once
it reaches recommended status. In addition to that, SQL language features of underlying
databases will be made available, when possible. Some capabilities will be estricted if a
database does not support them, such as subqueries which do not perform well in a dis-
tributeddatabasesystem.
5.2.3 Returndataformats
As dbserv will be TAP compliant, it will support the VOTable output specified by TAP 1.1, which
should be version 1.3. Additional encoding protocols for VOTable output will be supported,
includingJSONandSQLiteoutput,whichwerechosenfortheabilityto epresentmetadatain-
line with VOTable data. For all output formats, gzip compression through the HTTP protocol
willbesupportedandencouraged.
5.3 Image data access
Image discovery, metadata, and data etrieval will be available through the imgserv endpoint.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
29

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
5.3.1 Image finding
Image discovery will be facilitated through an implementation of the IVOA SIA V2 interface.
Simple implementations of SIA allow direct access to files available via HTTP by adding a link
to the SIA response; imgserv will use that attribute in the SIA response to link to another ser-
vice. ImagemetadatawillalsobefacilitatedthroughTAP,asimagemetadatawillbeavailable
according in the database through implementations of the IVOA ObsCore Data Model [15]
and the CAOM2 data model, as well as LSST-native image metadata epresentations.
5.3.2 Image etrieval
Images may be etrieved via the URL eturned in an SIA request, in the case images can be
directlyservedoverHTTP,orviaaSODAimplementation.TheLSSTSODAimplementationwill
alsoofferenhancementstoimplementproject-specificgoalswithoutpresentingadditional
interfaces to science users. For example, key/value pairs as used by the Data Butler client
maybeprovided.
5.4 Metadataaccess
Catalog and image metadata can be accessed via TAP query to provided metadata tables.
5.4.1 DataReleases
Each Data Release has its own set of tables. These are expected to differ in schema between
Data Releases. In addition, the image and catalog metadata available for each Data Release
willvary.Finally,identifierswithincatalogsandimages(exceptrawimageidentifiers)willvary
betweenDataReleases.
Accordingly, the Data Release number is a key parameter that must be presented to the Web
APIs. Collections of datasets that are not part of a Data Release (e.g. in-process, unreleased
data or intermediate data products) are assigned labels that are used in place of the Data
Release number.
The number of Data Releases available through the LSP and therefore the Web APIs aspect is
discussedinsection2.2.8,
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
30

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
5.4.2 Tables
AvailabletablesincludethoseinLDM-153.
Catalogandimagemetadatatableswillbeimplementedthrough IVOAandcommunitystan-
dards,whenpossible,suchasthoselaidoutby IVOA’sObsCoreDM,VOResource/RegTap,and
theCAOM2datamodel. Evenifdirectusageofthosemodelsisnotfeasible,abest-effort
export to them will be performed. In such a case, the "native" metadta tables will be docu-
mentedfordirectaccessbyusersthroughTAP.
5.4.3 TableStructure
Each table will have associated metadata that ecords, for each column, a description, UCD
when appropriate, unit when appropriate, and any relationship with other columns (e.g. so
severalcolumnsmakingupacovariancematrixcanbeidentifiedassuchwithout esortingto
pattern-matchingofcolumnnames).
5.5 UserWorkspaceaccess
WebDAVand/orVOSpacewillbeprovidedforuserworkspaceaccess.

Back to top


6 The InterconnectednessoftheSciencePlatform
A key element in the Science Platform design is that by layering the three Aspects on top of
the same underlying services, it enables scientific workflows that cross from one Aspect to
another.
For example, work can begin with using the Portal Aspect to find data, continue in the Note-
book Aspect to analyze it and create and store a derived data product, and the API aspect to
access it remotely. Or, a complex query can be constructed in the Notebook Aspect and the
resultssentforvisualizationinthePortalAspect.
The ability to work in this manner depends on three underlying capabilities of the Science
Platform.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
31

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
6.1 Sharing Data
All three Aspects expose the same LSST data products and provide access to the same user
workspaceanduserdatabasedata.Theuserworkspace,likelytobeaccessibleviaaVOSpace
service, is available in any place in the Portal where the download or upload of data is seman-
ticallyappropriate.Forinstance,atargetlistfromanothersurveyormissioncanbeuploaded
to the workspace, accessed in the Portal to perform a cross-match query, and analyzed in the
NotebooktogetherwithotherLSSTdatatoidentifystructureinthecombineddata.
6.2 SharingQueries
BecausetheunderlyingDAXservicesallowforasynchronousqueries,withaquerysubmitted
and assigned a name which is available for later re-use, and because the system is capable
of ecordingthehistoryofqueriesforauser,aquerycanbesubmittedfromoneAspectand
theresultsaccessedfromanother.
Queries can be constructed using the Portal UI, with the Portal’s visualization tools used for
a quick look to ensure that the results are as expected, and then the query results can be
etrieved from the Notebook - or the query itself can be programmatically repeated at a later
date, e.g., by someone tracking the development of Level 1 data on a transient object.
Or complex queries can be programmatically constructed in ADQL from Python code in the
Notebook,andtheresultsdiscoveredandbrowsedinthePortal.
6.3 Sharing State
Finally, the Portal exposes both read and write access to elements of its internal state. User
code can push data at the Portal for visualization using its Python or Web APIs, or modify the
stateofadisplay(e.g.,animagestretch).Thiscapabilitycanbeusedforoverlays;forexample,
if the user is viewing an image in the Portal and has identified points or regions of interest
in the image using code in the Notebook, those coordinates can be pushed to the Portal as
anoverlay. Usercodecanalso etrievestatefromthePortal;thiscanbeusedto etrieve
data from a user’s session, but also to facilitate interactions with the data. For example, the
position of a point or region selection on an image, or a row selection on a table, that the user
has made in the Portal UI can be made available to user Python code in the Notebook.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
32

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07

Back to top


7 ApplicationoftheSciencePlatforminsidetheProject
The Science Platform is not only for use by scientist end users through the Data Access Cen-
ters but it also forms an integral part of the toolchain within the Project. A phased release of
capabilities focuses on the needs of the primary target audience for each major version. This
approach allows us to deliver features of the platform to project staff to assist them in con-
truction work while also providing an early user base whose feedback will allow us to refine
ourdesign.
7.1 DataManagementTeams
The earliest releases of the Science Platform are aimed at Data Management teams. The
capabilitiessupportingsoftwaredevelopmentincludeinteractivelytroubleshootingcodeand
analysingmetricsthroughtheNotebookAspect.Thecapabilitiessupportingalgorithmicde-
velopmentincludevisualisingtheresultsoflargeprocessingrunsofprecursorandsimulated
datathroughthePortalAspect.
7.2 CommissioningTeam
Intermediate releases of the Science Platform are aimed at the commissioning team. The
capabilitiessupportingcommissioningincludetheinterfacesneededtoallowNotebooksto
cross-correlateearlyComCamdatawithtelemetrycapturedintheEngineeringandFacilities
DatabaseandadeploymentoftheSciencePlatformontheCommissioningCluster.
7.3 ScienceValidationTeam
MaturereleasesoftheSciencePlatformareaimedattheScienceValidationTeamwhichisex-
pectedtoinvolveamixtureofProjectandScienceCollaborationmembers.ScienceValidation
equires many of the features of the operational Science Platform since it involves detailed
interactivedataanalysisandgeneratingandvisualisingstatisticsfromlargedataruns.
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
33

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
7.4 ObservatoryOperations
In the Operations era, all of the above workflows will continue to be in heavy use by Obser-
vatorystaff:softwaresupportwillcontinuetocharacterisereleasesusingthenotebooksand
eportingtoolsdevelopedinConstruction;facilitycheckoutafterengineeringperiodsatthe
telescope will re-use many of the processes developed in commissioning; and facility scien-
tistsandcalibrationspecialistswilluseworkflowssimilartothosedevelopedduringscience
validation to enhance the understanding of the instrument. This will ensure that all Science
PlatformAspectsmadeavailabletothecommunityremainvigorousandrelevant.

Back to top


References
[1] Common Archive Observation Model, URL
http://www.opencadc.org/caom2/
[2] Comunidad Federada REUNA, URL
http://cofre.reuna.cl/index.php/en/
[3] InCommon:Security,PrivacyandTrustfortheResearchandEducationCommunity,URL
https://www.incommon.org/
[4]
[LDM-153]
, Becla, J., 2013,
LSST Database Baseline Schema
, LDM-153, URL
https://ls.
st/LDM-153
[5]
[LDM-554]
, Ciardi, D., Dubois-Felsmann, G., 2017,
Science Platform Requirements
, LDM-
554, URL
https://ls.st/LDM-554
[6] Dowler,P.,2012, In:Ballester,P.,Egret,D.,Lorente,N.P.F.(eds.)AstronomicalDataAnaly-
sis Software and Systems XXI, vol. 461 of Astronomical Society of the Pacific Conference
Series, 339,
ADS Link
[7] Dowler, P., Rixon, G., Tody, D., 2010, Table Access Protocol Version 1.0, IVOA Recommen-
dation 27 March 2010
[8] Dowler, P., Bonnarel, F., Tody, D., 2015, IVOA Simple Image Access Version 2.0, IVOA
Recommendation23December2015
[9]
[SQR-018]
,Economou,F.,2017,
InvestigationsintoJupyterLabasabasisfortheLSSTScience
Platform
, SQR-018, URL
https://sqr-018.lsst.io
[10] Graham, M., Morris, D., Rixon, G., et al., 2013, VOSpace specification Version 2.0, IVOA
Recommendation29March2013
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
34

Draft
LARGESYNOPTICSURVEYTELESCOPE
SciencePlatformDesign
LDM-542
LatestRevision2017-07-07
[11]
[LSE-319]
, Jurić, M., Ciardi, D., Dubois-Felsmann, G., 2017,
LSST Science Platform Vision
Document
, LSE-319, URL
https://ls.st/LSE-319
[12]
[LSE-163]
, Jurić, M., etal., 2017,
LSSTDataProductsDefinitionDocument
, LSE-163, URL
https://ls.st/LSE-163
[13] Osuna, P., Ortiz, I., Lusted, J., et al., 2008, IVOA Astronomical Data Query Language Ver-
sion2.00, IVOARecommendation30October2008
[14]
[LPM-122]
, Petravick, D., 2015,
LSST Information Classification Policy
, LPM-122, URL
https://ls.st/LPM-122
[15] Tody, D., Micol, A., Durand, D., et al., 2011, Observation Data Model Core Components,
its Implementation in the Table Access Protocol Version 1.0, IVOA Recommendation 28
October2011
DRAFTNOTYETAPPROVED
– Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
LSSTDMTechnicalControlTeam. –
DRAFTNOTYETAPPROVED
35

Back to top