1. Data Management Extended Monthly Report
  2. February 2015
    1. NCSA / University of Illinois
    2. Current accomplishments:
    3. University of Washington
    4. Princeton University and University of California, Davis
    5. IPAC / California Institute of Technology
    6. SLAC / Stanford University
      1. 02C.06.02.04 Image Service
      2. · Finished image cutout service V1
      3. · 02C.06.02.04 Image Service
      4. · Start work on ImageServ v2 (DM-1702) – implement image stitching
    7. LSST Program Office

Data Management Extended Monthly Report

Back to top

February 2015

NCSA / University of Illinois

Current accomplishments:

02C.07.00 Processing Control and Site Infrastructure Management

AURA is creating draft of amendments to the NCSA sub award for procurements, will review and comment when received.

Completed S15 cycle plans. Two DM members from NCSA traveled to SLAC for the DM All-Hands meeting mid-month; this meeting guided our planning within the larger picture of LSST-DM goals.

Completed a draft of the base site data center requirements. One more round of edits to fill in remaining TBDs is needed.

02C.07.01 Processing Control

Developed a handler to emit events for log messages using the Apache log4cxx logging framework.

02C.07.02 Infrastructure Services

Started defining our “unit of data management” and policies for usage thresholds and enforcing quotas.

02C.07.03 Environment and Tools

Implemented use of Python scripts to automatically launch an OpenStack instance and install LSST software will be explored.  

02C.07.04 Site Infrastructure  

Configuration continued and performance tests conducted using various operating systems.

Set up a network and bastion host for IPMI and documented how it can be used with the LSST infrastructure.

Planned activities:

02C.07.00 Processing Control and Site Infrastructure Management

Complete amendmentsto the NCSA sub award for procurements.

Execute on S15 cycle plans.

Complete base site data center requirements.

02C.07.01 Processing Control

Another iteration of feedback will confirm whether this version of the state diagram of the nightly compute simulator is sufficient or whether another draft will be constructed in the coming month.

02C.07.02 Infrastructure Services

Complete definition of our “unit of data management” and policies for usage thresholds and enforcing quotas.

02C.07.03 Environment and Tools

Continue OpenStack evaluation with focus on automation.

02C.07.04 Site Infrastructure  

Start deployment of the servers.

University of Washington

Current accomplishments:

02C.03.00 -- Alert Production Management Engineering and Integration

Simon attended the DESC meeting the first week of Feb. Russell and Simon attended the DM all-hands meeting. Both meetings were at SLAC. At the DM AHM we planned the Summer 2015 development cycle. Simon conducted several video conference interviews with candidates for the Research Scientist positions. UW hosted a candidate for the Research Scientist position the week of Feb. 16.

02C.03.08 -- Astrometric Calibration Pipeline

The astrometric solver can now be written to use files other than astrometry.net index files. The current default reference catalog still uses index files behind the scenes, but with advancements in the works from the DB team, we will be able to replace index files with a less opaque indexing scheme as the default. Russell spent time standardizing the reference catalog formats and fixing up tasks that use the reference catalogs to all be consistent.

Planned activities:

02C.03.00 -- Alert Production Management Engineering and Integration

Simon will host another candidate for the Research Scientist job. UW will also host Erik Tollerud of Astropy to talk about synergy and philosophy of the two projects. Simon will also be traveling to the Open Science Grid All-Hands meeting to present on LSST.

02C.03.05 -- Application Framework for Exposures

Russell will be working on sanitizing the interface to C++ methods so that numpy types behave more naturally.

02C.03.04 -- Image Differencing Pipeline

Simon will work on cleaning up the image differencing pipelines to be more in line with modern APIs and frameworks.

Princeton University and University of California, Davis

This report covers work carried out in FM5 of FY15 in the Data Release Production group (staff at Princeton plus Price and Gee working remotely).

Current accomplishments:

02C.04.00 Data Release Production Management Engineering and Integration


Significant effort was spent on planning over the course of this month. This was focused around both the Data Management All Hands Meeting, which took place towards the middle of the month, and the change of cycle (from Winter to Summer 2015) scheduled for the end of the month.

Together with Krughoff (UW), Bosch & Swinbank initiated a major effort to rethink the software development roadmap, LDM-240, in the light of both practical experience and the new staff that have become available over the previous months. This task has subsequently been widened and taken up by the rest of DM. A revised version of the DRP plan for FY15 was made available by the end of this month. This activity will continue into March.

The All Hands meeting provided a focus for planning our work in S15. Discussions there highlighted the particular importance of closer integration with the Hyper Suprime Cam system which we are relying on to provide test data. Our S15 plans were updated to match, and loaded into PMCS by the end of the cycle.

02C.04.01 - Application Framework for Catalogs

The skeleton of the calibration and ingest system, which transforms raw measurements (e.g. pixel positions) to calibrated quantities (e.g. celestial positions) suitable for database ingest was completed and incorporated in the LSST stack. Future work in this area will include developing specific calibration transformations to handle the different quantities being measured. [DM-1074]

The core parts of the transition of the afw::table system to the new measurement framework was completed. [DM-1099]

02C.04.03 - PSF Estimation

No work was carried out under this WBS during this month.

02C.04.06 - Object Characterization Pipeline

Basic processing of Hyper Suprime Cam data through the LSST pipeline is now possible. However, there remain a number of problems with pixel units and scaling which are still under investigation. [DM-1573]

The overhaul of the measurement framework was completed and the bulk of our measurement algorithms were converted to run in the new framework. This represents the culmination of much of the work tackled under this WBS throughout this cycle. [DM-1099, DM-1100]

Planned activities:

Continue LDM-240 re-planning with an aim to revise the complete document by the end of the month.

Post-transition improvements to some measurement algorithms and the framework are required in light of the experience gained in the transition. Resolving these is our first priority in S15. [DM-1769]

Resolve remaining issues in processing Hyper Suprime Cam data with the LSST stack and start incorporating the improvements which have been implemented on HSC, in particular the multi-band deblender. [DM-1907]

Re-start work on the upgrades to the Footprint system, in particular focusing on the design of the new API. [DM-1904]

Begin detailed planning of the measurements required in the galaxy sheer fitting study, which is a major deliverable for S15. [DM-1108]

IPAC / California Institute of Technology

Current accomplishments:

The IPAC group accomplishments included:

·   02C.05.00 Attended the DM AHM at SLAC

·   02C.05.00 LDM-240 road map revamp for Y15

·   02C.05.00 Finished S15 epics planning

·   02C.05.00 Continue the discussion of data access APIs with SLAC group

·   02C.05.01 Made the decision on JavaScript package adoption, React

·   02C.05.02 Visualization discussion in SUI team, David Ciardi led a discussion at the DM AHM

·   02C.05.05 Workspace discusion in SUI team, Xiuqin Wu led a discussion at the DM AHM

Planned activities:

·   02C.05.00 weekly SUI requirement and design discussions

·   02C.05.00 LDM-240 road map revamp for Y16-Y20

·   02C.05.00 Continue the discussion of data access APIs with SLAC group

·   02C.05.01 Made the decision to use React JavaScript package, restructured the source repository to include JavaScript code

·   02C.05.02 Visualization discussion with Camera team, meet with Tony Johnson

·   02C.05.02 Start working on the Python APIs to interact with Firefly visualization components

SLAC / Stanford University

Current accomplishments:

02C.06.00 Science Data Archive and Application Services Management Engineering and Integration

·  Orchestrated DM AHM @ SLAC (local logistics)

·  Finished planning database team work for the Summer 2015 cycle

·  Organized weekly Qserv and Data Access meetings

·  Continued preparing for XLDB 2015 Conference, initiated discussions about XLDB-for-government conference in D.C.

·  SLAC related: Delt with issues related to SLAC reorganization (accounting, support, office space)

02C.06.01.01 Catalogs Alerts and Metadata

·   Finished building 1st Metadata Store prototype

·   Prepared for and delivered Web Data Access Services demo at DM-AHM

02C.06.01.02 Image and File Archive

· Finished front-end of the form for collecting user input data
02C.06.02.01 Data Access Client Framework
· Butler V 2: started understanding inheritability and reusability of dataset types
02C.06.02.03 Query Services
· Finished planned Qserv refactoring work (DM-1030)
· Finished planned w ork on Management of distributed Qserv databases and tables (DM-1 036)
· Finished implementation of C++ geometry and porting to Qserv (DM-214)
· Finished planned work on integration testing (DM-1031)
· Improved Qserv stability (DM-1061)
· Released Qserv 201502 and accompanied documentation

02C.06.02.04 Image Service

·   Finished image cutout service V1

Planned activities:

02C.06.00 Science Data Archive and Application Services Management Engineering and Integration

·  Organize weekly Qserv and Data Access meetings

·  Participate in DM T/CAM training session

·  Present (keynote) at the Tools for Astronomical Big Data workshop

·  Revisit long term database plan

·  Search for candidates for remaining open position

·  Continue preparing for XLDB 2015 Conference

02C.06.01.01 Catalogs Alerts and Metadata

·   Implement Data Ingest (DM-210) – moved from W15 cycle

·   Start work on MetaServ v2 (add support for DDL, define JSON results)

02C.06.01.02 Image and File Archive

·   No plans in this area in March

02C.06.02.01 Data Access Client Framework

·   No plans in this area in March

02C.06.02.03 Query Services

·   Start designing data distribution and replication (DM-1060)

·   Start work on Qserv Refactoring (DM-1707) – unit testing, query killing, duplicate column issue

·   Start work on DM & Table Mgmt (DM-1703) – worker mgmt service

·   Start work on Multi-node Multi-query integration testing harness

·   (DM-1948)

·   Start work on Release Qserv 201503 with documentation – research, worker side squashing

·   02C.06.02.04 Image Service

·   Start work on ImageServ v2 (DM-1702) – implement image stitching

LSST Program Office

DM Project Management and Control

Current accomplishments:

The DM Project Manager:

·   Continued to support NSF Cooperative Agreement risk/contingency negotiations.   Submitted risk correlations for DM plan in PMCS and 9 upscopes for LPM-72 update.
·   Interviewed and hired Alex Withers of NCSA as the Information Security Officer for LSST, implementing the key position in the IT Security Program (aka Cyber-security plan). Alex will work 25% on LSST.
·   Continued recruiting and hiring activities: filled the DM Deputy System Architect (internal posting) and DM SQuaRE Software/Documentation Engineer positions.
·   Continued work with AURA on MREFC contracts for all DM lead institutions. The IPAC, SLAC, Princeton, NCSA, UW, and REUNA MREFC contracts are signed. The FIU and UCD sub award contracts have completed final round of comments with AURA contracts and institution counterparts and are awaiting signature. The total value of these contracts is $102M, with the FY15 authorization a total of $9.8M.
·   The Memorandum of Agreement with IN2P3 for data processing operations was completed and is ready for anticipated signature on March 5 in an executive meeting at IN2P3.
·   Continued tracking progress of the MOA for Brazilian contributions to operations (Networks). The MOA has been briefed to the responsible agencies, the Ministry of Science and Technology (MCTI) and the Sao Paolo Research Foundation (FAPESP). A letter of support enabling the MOA and allowing for signature by the Brazilian RENs RNP and ANSP is in process.
·   Completed the Summer 2015 detailed plan.
·   Attended and led several sessions at the DM All Hands Meeting at SLAC.
·   Planned the next Data Management All Hands Meeting at NCSA May 18 - 20.
·   Participated in the feedback session at the AURA Board meeting.
·   Briefed DM status at the LSST Corporation Board meeting.

Planned activities:

The DM Project Manager will:

·   Continue to support NSF Cooperative Agreement Risk/Contingency negotiations.
·   Continue recruiting and hiring: open DM SQuaRE technical position.
·   Complete work with AURA on MREFC contracts for FIU contract, and amendment to NCSA contract covering LSST equipment procurements.
·   Continue supporting process to move draft Memoranda of Agreement with Brazil to signature.
·   Conduct Technical/Cost Account Manager (T/CAM) training in Tucson on March 25 - 26, covering PMCS, Risk Management, Travel and Administrative Systems, and JIRA Agile/PMCS/LDM-240 integration.
·   Participate as panel member in the Square Kilometer Array Science Data Processing Preliminary Design Review at Jodrell Bank March 18 - 19.
·   Conduct T/CAM training in Tucson March 25 - 26.

DM Science

Current accomplishments:
Staffing up continued to dominate February activities, in addition to travel to support DM and project-wide activities such as the Project Science Team meeting, the DM All-Hands Meeting and the Dark Energy Science Collaboration meeting.
The position for DM SQuaRE Science lead remains unfilled. The candidate to whom the offer has been made has turned it down (due to a competing, tenure-track, counter-offer from their current institution). Another prospective candidate has withdrawn from consideration (again, after a pre-emptive counter-offer by their current institution). The search committee is discussing whether to interview more candidates, or develop alternative strategies to ensure adequate scientific leadership.
An offer has been made for the DM Analyst scientist postdoc position (renamed to LSST Data Systems Fellow) to Colin Slater. Colin accepted, and is expected to begin in September 2015. Unfortunately, the attempt to hire two people into that position failed because sufficient non-project funds were not available to cover both hires.
The DM Project Scientist took part in the PST meeting at SLAC, as well as presented a talk on LSST DM at the DESC meeting, and also participated at the first DM All Hands meeting at SLAC, contributing to a variety of discussions.
Planned activities:
In March, the DM Project Scientist will focus on completion of hiring activities, as well as increasingly more activity in updating the key science/technical baseline documents.
DM System Engineering

Current accomplishments:

Activities completed by the DM System Architect include:

·   Helped conduct DM All Hands Meeting.
·  Started refining Base Center design.
·  Discussed status of Sizing Model and noted action items.
·  Refreshed team on development practices and tools.
·  Assisted with deep dive into Data Release algorithms and dataflows.
·   Helped with design and implementation of REST API for data access services.
·   Commented on and resolved many RFCs.
·  Coding standards updates.
·  Package interface changes.
·   Gave input to SUI design.
·   Gave input to Base Center design.
·   Helped with middleware planning for Summer2015 and beyond.
·   Provided DM-based use cases for Technical Operations Working Group.
·   Interviewed and selected candidate for Deputy System Architect position.
·   Attended OCS Middleware Workshop to define details of OCS communication.
·   Worked to define details of Camera DAQ protocols (LSE-68/69).
·   Delivered talk on LSST data movement at CrossConnects Cosmology workshop.
·   Participated in Chilean Networks Acquisition review.
·   Provided information about speeds&feeds for NCSA Infrastructure hiring.
·   Fixed some problems in eups.
·   Continued work on New Butler.
·   Assisted with preparations for XLDB-2015.

Planned activities:

The DM System Architect will:

·   Update LDM-240 roadmap.
·   Interview candidates for SLAC position.
·   Continue work on Camera DAQ protocols (LSE-68/69).
·   Continue to give input to Base Center design.
·   Educate new Security Officer, new Deputy System Architect, SLAC group.
·   Define DM-related personnel for Technical Operations Working Group.

DM Science Quality and Reliability Engineering (SQuaRE)

Current Accomplishments:

02C.01.02 Management / LoE

·  Recruitment for Documentation Engineer position - offer made and accepted

·  DM All Hands meeting

·  S15 EVM Planning

·  JH attended the SCALE conference

02C.01.02.02 SDQA Toolkit aka PipeQA

·  A kickoff meeting exploring closer collaboration between this WBS and the SUI effort at IPAC was had

02C.01.02.03 L3 Toolkit aka Productized Stack

·  Documentation was updated for the Stack v10_0 release

02C.07.03.01 Software Development Tools aka Developer support

·  The migration of the DM primary software development to Github was carried out. Only some data-heavy repos remain on the Gitolite server

·  The JIRA Team component was re-organized to match the DM T/CAM structure

Planned Activities:

·  Release engineering activities / v11 (aka W15) release

DM Applications, Middleware, and Infrastructure

Current accomplishments:

·  Completed Winter 2015 applications and middleware and closed out stories and epics in JIRA, ready for start of Summer 2015.

·  The LSST Chilean Networks Acquisition Review was conducted and passed on February 24. The review confirmed the overall Chilean networks acquisition plan as a series of 5 contracts with a total value of $9.6M. The panel recommended we defer implementing path diversity on Mountain - Base link as a possible upscope.

Planned activities:

·  Execute Summer 2015 plan for March.
·  Complete amendment to NCSA contract for first acquisitions of Development and Integration clusters.
·  Complete Base Data Center requirements and prepare statement of work for design consultant to support AE contract.

Back to top