1. Project Status
  2. Current Photos
  3. Risk Management
  4. Detailed Project Progress and Status

Data Management Summary Monthly Report

March 2015
 
Project Status

The Summer 2015 release started, with all teams working on the next round of features and tests against the LDM-240 Key Progress Metrics. Highlights of this release include:

·  Aperture correction measurement and error estimation (UW)

·  Programmatic interface for astrometric self calibration (UW)

·  Sky template building strategy (UW)

·  Multi-epoch astrometric solutions (UW)

·  Distributed database demonstration at 10% scale - 100+ TB, on 50 nodes (SLAC)

·  Data distribution and replication prototype (SLAC)

·  Galaxy fitting – shear precision experiments (Princeton)

·  Programmatic interface for full focal plane PSF estimation (Princeton)

·  Alert production simulator (NCSA)

·  Catalog and image interactive visualization (IPAC)

·  Level 3 science pipeline toolkit requirements, tool evaluation (SQuaRE, IPAC)

·  Simple orchestration services (NCSA)

·  Dynamic hardware provisioning – Openstack (NCSA)

·  Base Data Center design (NCSA)

·  Continuous Integration of software (SQuaRE)


The Data Management Development Roadmap (LDM-240), which is the primary "measuring stick" of DM development progress, was updated for the remainder of FY15, and a more extensive update covering FY16 - FY18 was initiated. This "bottom-up" iteration will for the first time incorporate input from the institutional Technical/Control Account Managers (T/CAMs) and will align the Roadmap with their detailed plans for development in each location.
 
The IPAC, SLAC, Princeton, NCSA, UCD, UW, and REUNA MREFC contracts are signed. The FIU sub award contract has completed final round of comments with AURA contracts and institution counterparts and is awaiting signature. The total value of these contracts is $102M, with the FY15 authorization a total of  $9.8M.
 
The amendment to the NCSA agreement for the procurement of hardware was drafted. The Memorandum of Agreement with IN2P3 for data processing operations was signed on March 5 in an executive meeting at IN2P3. The Brazil MOA (Networks) has been briefed to the LSSTC committee on international partnerships and is in revision based on their comments.
 
Recruiting and hiring activities continued across all DM institutions: seven interviews were conducted, five offers were made. Fifteen positions have been filled to date since the MREFC award, while nine positions are currently open.
 

Back to top



Current Photos
 
No new photos this month.
 

Back to top



Risk Management
 
The DM Project Scientist had a productive working two-day meeting with Eric Tollerud (AstroPy coordinator) on understanding the architecture and opportunities for collaborations with the AstroPy project. The key results have been discussed with the DM System Architect. They resulted in an increase of probability of occurrence of DM-065 risk ("LSST DM software architecture incompatible with de-facto community standards") from 25-50% to 50-75% due to divergence between DM's current C++-centric model and the pure Python model of AstroPy. Mitigations will be analyzed to determine the level of incremental effort needed to more closely align with this emerging de facto standard.

 

Back to top



Detailed Project Progress and Status

DM Project Management and Control

Current accomplishments:

The DM Project Manager:


·   Continued recruiting and hiring activities. We made an offer for the DM SQuaRE Scientist position.
·   Continued work with AURA on MREFC contracts for all DM lead institutions. The IPAC, SLAC, Princeton, NCSA, UW, UCD, and REUNA MREFC contracts are signed. The FIU sub award contract has completed final round of comments with AURA contracts and institution counterparts and is awaiting signature. The total value of these contracts is $102M, with the FY15 authorization a total of $9.8M.
·   The Memorandum of Agreement with IN2P3 for data processing operations was signed on March 5 in an executive meeting at IN2P3.
·   Continued tracking progress of the MOA for Brazilian contributions to operations (Networks). The MOA has been briefed to the LSSTC committee on international partnerships and is in revision based on their comments.
·   Continued planning and coordination for the next Data Management Leadership Team Meeting at NCSA May 18 - 21.
·   Prepared briefing on DM status, planning process, and upcoming plans for the AMCL April meeting. Total of 100 slides prepared covering all aspects of DM.
·   Conducted Technical/Cost Account Manager (T/CAM) training in Tucson on March 25 - 26, covering PMCS, Risk Management, Travel and Administrative Systems, and JIRA Agile/PMCS/LDM-240 integration.
·   Participated as panel member in the Square Kilometer Array Science Data Processing Preliminary Design Review at Jodrell Bank March 18 - 19.
 

Planned activities:

The DM Project Manager will:


·   Continue recruiting and hiring.
·   Complete work with AURA on MREFC contracts for FIU contract, and the amendment to the NCSA contract covering LSST equipment procurements.
·   Continue supporting process to move draft Memorandum of Agreement with Brazil to signature.
·   Complete planning and coordination for the next Data Management All Hands Meeting at NCSA May 18 - 21.
·   Conduct T/CAM training in Tucson April 8 - 9.
·   Attend the LSST First Stone, AMCL, and LSST Corporation Board meetings in Chile April 13 - 17.
·   Represent LSST at the South American Astronomy Coordination Committee (SAACC) meeting in Chile April 20.

DM Science

Current accomplishments:
Staffing continued to be the focus of March activities, with interviews for Level 1 DM lead search at UW, and visits to DM group in Tucson.  Another promising candidate was interviewed for Level 1 DM science lead position at University of Washington. The position has informally been offered, and we're waiting to hear back whether the candidate is interested. Another offer has also been made for the SQuaRE science lead position in Tucson; we're awaiting for the response from that candidate as well.
 
The DM Project Scientists has begun work on the update of the Galactic star counts model for the LSST sizing model, with the help of a graduate student.
 
 The DM Project Scientist also had a productive working two-day meeting with Eric Tollerud (AstroPy coordinator) on understanding the architecture and opportunities for collaborations with the AstroPy project. The key results have been discussed with the DM System Architect. They resulted in an increase of probability of occurrence of DM-065 risk ("LSST DM software architecture incompatible with de-facto community standards") from 25-50% to 50-75% due to divergence between DM's current C++-centric model and the pure Python model of AstroPy.
 
Finally, the DM Project Scientist contributed patches to EUPS and the lsstsw build and continuous integration system, resulting in speed-ups of builds of approximately ~2.3x, and also contributed materials for the presentation on the state of DM for the April AMCL meeting.
 
Planned activities:
 
In April, the DM Project Scientist expects a continued focus on completion of hiring activities, as well as increasingly more activity in updating the key science/technical baseline documents.
 
DM System Engineering

Current accomplishments:

Activities completed by the DM System Architect include:


·  Updated LDM-240 roadmap.
·  Interviewed candidates for SLAC position.
·  Continued work on Camera DAQ protocols (LSE-68/69).
·  Continued to give input to Base Center design.
·  Educated new Security Officer, new Deputy System Architect, SLAC group.
·  Defined DM-related personnel for Technical Operations Working Group.

 
Planned activities:

The DM System Architect will:


·  Complete LDM-240 updates and review document
·  Take vacation

 
The DM System Interfaces Scientist will:

·   Produce drafts for CCB review of revisions to LSE-68, LSE-69, and LSE-75.
·   Produce description of summit-base-link-severed use cases for the Technical Operations Working Group
·   Analyze the DM-related dependencies in the PMCS during the Commissioning period and ensure that they are up to date, and synchronized with corresponding Camera integration activities.

 
DM Science Quality and Reliability Engineering (SQuaRE)
 

Current Accomplishments:

02C.01.02
The W15 stack release (v10_1) process started. Two release candidates were prepared in March. The release was blocked on a number of bugs that spilled into April. A full release report will be provided next month.

02C.01.02.04

A new system was released for beta-testing that allows for the automation of clean-slate builds on a variety of operating systems provided by cloud back ends. It is based on Vagrant and Puppet and eventually will support a number of APIs including OpenStack, AWS, and Digital Ocean.

The beta testing version supports Digital Ocean as the most available provider for now. This allows us with a few commands to have a brand new instance of, say, RedHat 7 to be provisioned with the stack pre-requisites, and verify the user-facing experience (via newinstall.sh from our distribution server). This will allow release testing to speed up considerably and for us to be able to verify the stack’s availability on a wider range of platforms.

Work started on a Jenkins-based CI system to replace buildbot. The new system will allow for an arbitrary number of build slaves to be be instantiated on demand from a cloud or virtualised back end. A proof of concept of running the stack build script in a Jenkins slave was completed.

Planned Activities:

·  Release engineering activities for v10_1 release

·  CI system to prototype stage

DM Applications, Middleware, and Infrastructure

Current accomplishments:


·  Executed Summer 2015 plan for March (refer to individual institution reports in DM Extended Monthly Progress Report for details).
·  Continued work on updating the DM Sizing Model unit quotes in preparation the first acquisitions of the Development and Integration clusters.
·  Completed Base Data Center requirements and prepared statement of work for design consultant to support AE bid process.
·  Completed work on making IPAC Firefly an open source package, accessible through Github. Firefly will be the foundation of the first generation Science User Interface. The SUI group has also conducted meetings to support visualization needs for SQuaRE and the Camera team.

·  DM team members participated in the LSST Technical Operations Working Group (TOWG). They developed use cases to help with specifications for the buildings in La Serena, justifying the required office space based on the necessary level of staff.

·  The base data center requirements were revised with updates to square footage, average rack weights, and power requirements. These revisions were reviewed with the Chilean site team.

Planned activities:


·  Execute the Summer 2015 plan for April (refer to individual institution reports in DM Extended Monthly Progress Report for details).
·  Complete the update to the DM Development Roadmap (LDM-240).
·  Complete the DM Sizing Model update.

Back to top