1. Project Status
  2. Current Photos
  3. Risk Management
  4. Detailed Project Progress and Status
    1. LSST Program Office
    2. University of Washington
    3. Princeton University and University of California, Davis
    4. IPAC / California Institute of Technology
    5. SLAC / Stanford University
    6. NCSA / University of Illinois
    7. Current accomplishments:
    8. AURA/LSST and AURA/NOAO
    9. Current accomplishments:

Data Management Monthly Report

November 2015
 
Project Status


·  The Winter 2016 release work continued and remains on track to complete in February, 2016. Features implemented this month include:
o   Back-ported HSC-developed parallelization code, including job submission and inter-process MPI messaging, as LSST ctrl_pool package. This is expected to be a stopgap until process execution middleware with layering of functionality is available.
o   Changed the defaults for creating coaddTempExps (single-visit images warped and trimmed to a coadd patch) to subtract the background and use a lanczos3 kernel.
o   Improved overscan correction for DECam data to handle "bias jumps". Other ISR improvements are ongoing; "glowing edges" remain to be fixed.
o   Added fixes throughout the stack for Mac OS X El Capitan compatibility including workarounds for security constraints and upgraded ActiveMQ messaging; documented3 what had to be done and why
o   Enabled obs_decam to be used to run ISR (but not crosstalk) with updated gain, read noise, saturation, and defects
o   Made the DECam-supporting camera-specific package ( obs_decam2 ) an official LSST product (though it is still under development)
o   Delivered initial prototype of new pipeline construction component ("SuperTask") and associated command-line wrapper ("Activator")
o   A framework is now available for injecting artificial sources into image data for test purposes.
o  A significant series of updates to the Hirata-Seljak-Mandelbaum shape measurement code (shapeHSM) were ported from Hyper Suprime-Cam. This included corrections to masking, scaling of errors, and better unit tests.
o  Continued working on the conversion of Firefly client side software from GWT based code to React/FLUX based pure JavaScript code and provided support for Camera team to use Firefly to display the focal plane image
 
·  The team performed a differential chromatic refraction literature overview and discussed ways forward.
·  NCSA held two-day meeting of the Joint Coordination Committee with CC-IN2P3, including a special HPC facility coordination meeting involving representatives from Argonne, NERSC, and Blue Waters.
·  Continued design support and implementation of development and operations infrastructure
o   Decommissioned old servers.
o   Upgraded ActiveMQ message broker to 5.12.1.
o   Fixed HealPix-based skymaps.
o   Increased quotas on Nebula OpenStack cluster to accommodate increased usage.
o   Supported detailed design of La Serena data center.
o  Continued detailed design of Level 1 systems, including investigation of possibility of elimination of the network outage buffer.
·  SQuaRE released a “technote” rapid-development continuously deployed documentation platform
·  Released Qserv 2015_10 ( release notes ). Qserv is now released in standard manner and published by SQuaRE instead of Qserv team
·  Continued development of the contracts for the Chilean National, International, and Mountain – Base fiber optic networks and corresponding updates to schedules in PMCS.

·  Continued work on the NCSA contract amendment to fund the FY16 Infrastructure Acquisition Strategy.
·  Continued development of FY16 contract amendments for all DM partners. Amendments were executed with IPAC, NCSA, Princeton, SLAC, and UW. The amendment with UCD has been sent for signature. Smaller contracts for temporary contract staff at BNL, U Penn, and IN2P3 are also in work.
·  The DM Project Scientist continued to be focused on the study of the efficiency of LSST as a detector of NEOs, as well as participation in a series of meetings.

·  The Project Engineer and Infrastrucure Lead supported the operations proposal, including a draft WBS for operations.
·  The DM team submitted several papers at ADASS.
·  Recruiting and hiring activities continued across all DM institutions. Twenty-nine positions have been filled to date since the MREFC award, while 8 positions are currently open.

Back to top



Current Photos
 
There are no new photos this month.
 
 

Back to top



Risk Management
 
The DM Risk Register was reviewed in the monthly process. Significant changes were:
 

·   As of 7/28/15 there were 87 active DM risks with:
o   total expected FY2013 USD exposure cost of $14,799K
o   total expected then-year USD exposure cost of $17,575K
·   As of 11/23/15 there were 86 active DM risks with:
o   total expected FY2013 USD exposure cost of $17,409k
o   total expected then-year USD exposure cost of $20,618K
·   1 risk was realized: DM-041 16 bits too small to capture the dynamic range of a full-well. Exposure $76k/$93k
·   1 risk was retired: DM-024 Workflow System Doesn’t scale. Exposure $14k/$16k
·   One risk was added DM-256 Calibration Products Maturity. Exposure $122k/$130k

 

 

Back to top



Detailed Project Progress and Status


LSST Program Office

DM Project Management and Control

Current accomplishments:

The DM Project Manager:


·   Participated in the DM Leadership Team Meeting in Princeton.
·   Continued development of the contracts for the Chilean National, International, and Mountain – Base fiber optic networks and corresponding updates to schedules in PMCS.
·   Continued management of the T/CAMs and overseeing plans, budgets, invoices, payments and reporting for Earned Value.
·   Continued work with NCSA on contract amendment to fund the FY16 Infrastructure Acquisition Strategy.
·   Continued development of FY16 contract amendments. Amendments were executed with IPAC, NCSA, Princeton, SLAC, and UW. The amendment with UCD has been sent for signature. Smaller contracts for temporary contract staff at BNL, U Penn, and IN2P3 are also in work.
·  Recruiting and hiring activities continued across all DM institutions. Twenty-nine positions have been filled to date since the MREFC award, while eight positions are currently open. In Tucson, there is an open SQuaRE Observatory Scientist position.
·  Prepared draft slides for the upcoming NSF Annual Status Review of LSST.

 
Planned activities:

The DM Project Manager will:


·   Participate in Joint Technical Meeting planning
·   Update draft slides for NSF Annual Review of LSST
·   Continue work with NCSA on contract amendment to fund the FY16 Infrastructure Acquisition Strategy.
·   Continue development of FY16 contract amendments
·   Oversee the incorporation of 385 Level 3 milestones from JIRA into the PMCS
·   Continue recruiting and hiring activities
·   Present a paper on LSST Lessons Learned to the Next Generation VLA Workshop at NRAO in Socorro, NM
·   Along with the DM Project Scientist and Project Engineer, and the UK LSST group, discuss pre-operations activities of that group

 

DM Science

Current accomplishments:
November DM PS activities continued to be focused on the study of the efficiency of LSST as a detector of NEOs, as well as participation in a series of meetings.
 
Specific work includes:
 
In October, we succeeded in running MOPS (the Moving Object Pipeline System) up to findTracklets. Since then we have gotten all MOPS functions working except  linkTracklets. While linkTracklets runs and successfully generates tracks for parts  of the dataset, it exhibits odd behavior which we're in the process of understanding. We’re also updating MOPS to  compile using PAL (Positional Astronomy Library) as opposed to Slalib, which it now does on both Mac OSX and Linux (lsst-dev). Current  test runs indicate that Slalib and PAL based MOPS builds do not produce the same  results when running on the same data. Lastly, there has been substantial progress on the python code used to link the MOPS functions into one 'run-all' script including, but not limited to: unittest implementation, output redirection to files based on the running function, and the ability to use configuration files to specify MOPS parameters. All MOPS work described above is being lead by Joachim Moeyens.
 
Colin Slater has further characterized the rates and structure of false positives on DECam images processed with the LSST pipeline. These data will be used to generate realistic simulated asteroid catalogs, for testing of the efficiency of MOPS. We have also begun work on exploring the tradeoffs of different algorihmic approaches for morphological star-galaxy separation. This work is expected to result in better quantification of how well LSST-delivered star-galaxy separation will perform, which will be used to better specify the requirements in the DPDD.
 
Mario Juric has completed and submitted the LSST Data Management reference paper, The paper has been submitted to proceedings of ADASS XXV, and is intended to become a standard reference for the LSST DM System, though first light. The paper is currently available at http://ls.st/juh and will be uploaded to astro-ph in December. MJ has also participated in a series of meetings: presenting LSST’s capability for Solar System Science at the Division of Planetary Sciences meeting (National Harbor, MD), the DM Leadership Team meeting in Princeton, and the NSF MPS OpenData workshop (Washington, D.C.)
 
Planned activities:
 
In December, the DMPS team will remain focused on the NEO studies (the finalization of functions for simulating the expected rates of false positives), investigation of morphological star galaxy separation, and participations in meetings (the NSF Cyberinfrastructure meeting in Washington, D.C, PST meeting in Tucson, t he WL systematics meeting in Davis).
 
DM System Engineering

Current accomplishments:

Accomplishments by the DM System Architect and Deputy System Architect include:

·   Set agenda for and conducted DMLT meeting, resulting in improved long-term roadmap and resource requirements.

·   Worked on top-down operations plans for LOPT, including defining organizational boundaries and working on resource requirements.

·   Gave feedback on design for Base Data Center.

·   Reviewed DM risks, updated 16.

·   Wrote up basic requirements for workflow system.

·   Analyzed "rerun" usage requirement for Butler.

·   Developed proposal for making progress on Astropy integration.

·   Started posting weekly highlights of DM accomplishments.

·   Described DM Stack licensing and its effects on others desiring to use DM code.

·   Presented Data Butler to DESC SSim group.

·   Responded to survey about use of virtual machines for processing.

·   Worked with Mario Juric to finalize both the ADASS proceedings papers.

·   Finalized and released modifications required to allow the “lsst_distrib” components of the DM software to build and run on Mac OS X El Capitan.

·   Wrote DM Tech Note DMTN-001 describing the problem and the associated fixes.

·   Attended (remotely) the DMLT meeting in Princeton.

·   Had discussions with Gregory Dubois-Felsmann and Matias Carrasco-Kind on the superTask environment.


Planned activities:

The DM System Architect and Deputy System Architect will:

  
 
· Assist with NCSA work on Level 1 Concept of Operations.
· Review consistency of LDM-151 Applications Design with long-term
· roadmap.
· Discuss Data Butler feature priorities and design high-priority features.
· Follow up other action items from DMLT meeting.
· Contribute to TOWG and LOPT operations planning.
· Review updates to OCS Middleware ICDs and other documents.
· Test and experiment with Supertask infrastructure.
· Have discussions on operations planning
DM Science Quality and Reliability Engineering (SQuaRE)
Current Accomplishments:
02C.01.02 SDQA 
· Science Verification Activities
Work continues with a number of people attempting to process their designated datasets with the stack. Minutes for the Verification weekly telecons describing ongoing progress can be found at http://ls.st/f43
· Development Activities
Improvements were made on SQuaRE’s Nebula-based developer infrastructure. Some loose ends were tied with the GitLFS deployment. Investigative work was done on possible ways to distribute CI-generated binaries, as well as using Docker containers as CI “slaves”. A straw-man design of the QA harness was developed. A “technote” rapid-development continuously deployed documentation platform was released.
· Non-Technical Activities
FE & DN attended the DMLT meeting.
Planned Activities:
02C.01.02 Science Quality and Reliability Engineering
· Verification dataset processing
· GitLFS test deployment wrap-up
· NCSA Nebula Openstack cluster evaluation (cont)
· Developer documentation
· Automated binary environment development
DM Applications, Middleware, and Infrastructure
Current accomplishments:
· Refer to by institution reports below
Planned activities:
· Refer to by institution reports below


University of Washington

Current accomplishments:

02C.03.00 Alert Production Management Engineering and Integration

David Reiss (DR) travelled to Liverpool to participate in the Gaia transients meeting.  Simon Krughoff (SK) attended the DMLT meeting at Princeton.

02C.03.01 Single Frame Processing

John Parejko (JP) worked on taking a census of LSST code to determine how WCS like objects are used in the stack.  This work is part of the process of determining requirements for the improved Wcs classes scheduled to be delivered this cycle.

02C.03.04 Image Differencing Pipeline

DR and Ian Sullivan (IS) finished up the literature search on DCR (DM-3700).  They produced a report containing summaries of the relevant papers. The report also contains a brief sensitivity analysis of DCR on several atmospheric parameters.  DR and IS set up a meeting for early December to gather project wide input on mitigation of DCR in the context of image differencing.  DR started looking at the current implementation of the dipole measurement code.

02C.03.05 Application Framework

Russell Owen (RO) worked on making the calibration task easier to use and to extend.

Planned Activities:

02C.03.00 Alert Production Management Engineering and Integration

An ad will be in the December AAS register.  We will restart the research faculty search in December as well.

02C.03.04 Image Differencing Pipeline

IS and DR will conduct the meeting to gather project information on DCR mitigation.  DR will work on dipole measurement code.

02C.03.05 RO and SK will work on the calibrate task refactor.

 


Princeton University and University of California, Davis
This report covers work carried out in FM2 of FY16 in the Data Release Production group.
 
Current accomplishments:
 
02C.04.00 - Data Release Production Management Engineering and Integration
 
Management activities during this month centered on the DM Leadership Team meeting, which was held in Princeton during the week starting 16 November. Bosch, Lupton and Swinbank were directly involved in this meeting, plus Swinbank and local admin support in Princeton coordinated logistics for the meeting. A full discussion of the topics covered by the DMLT is outside the scope of this report, but information on the agenda and other materials are available from the relevant page on Confluence.
 
The DMLT meeting served as a focal point and climax for the ongoing effort to revise the Software Development Roadmap (LDM-240) and DM Applications Design document (LDM-151), and substantial management effort was invested on this during the month. This included extensive discussion of the requirements and plan for DRP in general (Swinbank, Bosch), for the Calibration Products Pipeline (02C.04.02, Swinbank, Lupton) and of those aspects of the Single Frame Measurement Pipeline (02C.03.01) which are specific to annual data release processing. This last topic evolved in to a videoconferenced meeting involving multiple DM sites; the plan is that similar sessions will be held covering aspects of the DRP plan in the months ahead.
 
The DRP plan as presented at the DMLT meeting has been entered into the JIRA-DLP long-term planning project, resource loaded, and then imported and from there imported to PMCS as the new baseline. However, the planning process is not yet complete. In particular, over the next several months, we expect:
 


·  A detailed review of the plan as presented by Lupton, in his role as DM Algorithms Lead;
·  Further detailed design documentation produced by Bosch, following the model of his Measurement of Blended Objects in LSST document;
·   Minor revisions and clarifications to the Data Products Definition Document (LSE- 163);
·   Completion and baselining of a revised version of LDM-151.
 

Hiring As of the start of November, Peter Melchior has joined LSST as a scientist. He will be spending 50% of his time working on LSST, and 50% on WFIRST. Peter has extensive experience in weak lensing and DES data analysis. In late November we began reviewing applications for the ”Postdoctoral Positions in Software” which were advertised on the AAS Job Register in September and October. We have scheduled an initial round of interviews for early December. Following discussion at the DMLT, we expect to continue staffing up aggressively.
 
02C.04.01 - Application Framework for Catalogs
 
Effort continued to focus on re-syncing the LSST codebase with Hyper Suprime-Cam. To that end, functionality to define polygonal regions defining the valid area of CCDs was used to mask the vignetted regions of HSC CCDs. [DM-3259]
 
Functionality was added to the skymap package to perform spatial searches on patches and tracts. For example, it is now possible to look up patches and tracts containing specific celestial coordinates. [DM-3775]
 
A framework is now available for injecting artificial sources into image data for test purposes. [DM-3380] The --rerun option for command line tasks was ported from Hyper Suprime-Cam. This provides a convenient shorthand notation for specifying input and output data repositories when executing a task. [DM-3371]
 
The parallelization system was ported from Hyper Suprime-Cam. Although we do not expect this system to be deployed in production – it should ultimately be replaced by a modernized system later in construction – it provides the necessary basis both for further algorithmic development on LSST and for HSC to directly use the LSST stack in production. That, in turn, is necessary to enable future HSC development to directly benefit LSST. [DM-2983]
 
A series of minor bug xes and improvements were made across the stack. These included:

·  A memory leak was xed in afw::geom::polygon. [DM-4408].
·  Updates of table field names and configuration parameters to current stack conventions. [DM-3798]
·  Unit tests were updated to match recent changes to coadd creation. [DM-4391]
·  Minor fixes were made to support changes required for compatibility with the latest release of Mac OS X (‘El Capitan’). Note that the bulk of this compatibility work was not carried out by the DRP group. [DM-4360] 2
·  An error was fixed in converting between afw coordinates and the HEALPix convention. [DM-4387]
·  Minor fixes were made to support of the new (1.10) version of NumPy. [DM-4238]
·  Legacy fitsthumb code was removed from the obs_subaru camera interface package used for processing Hyper Suprime-Cam through the stack. [DM-4323]
·  Fixed an error in Footprint merges which could cause apparent dropouts of sources. [DM-2978]
·  Added an explicit check for catalog ordering when finding child sources in a SourceCatalog, thereby avoiding a potential failure condition. [DM-2976]
·  Improved globbing when displaying command line task configuration. [DM-4217]
·  Fixed an error which caused debugging information to be missing from some command line task failure modes. [DM-4218]
 

02C.04.02 - Calibration Products Pipeline
 
Lupton and Fisher-Levine travelled to Chile for initial tests of the Collimated Beam Projector at CTIO. These experimental runs were successful, and effort will now turn to understanding and analyzing the data collected.
 
02C.04.03 - PSF Estimation
 
No work was carried out in this WBS element in this month.
 
02C.04.04 - Image Coaddition Pipeline
 
No work was carried out in this WBS element in this month.
 
02C.04.05 - Object Detection and Deblending
 
No work was carried out in this WBS element in this month.
 
02C.04.06 - Object Characterization Pipeline
 
A number of minor improvements were made to the ProcessCcdDecam task in support of ongoing verification dataset work being coordinated by SQuaRE. [DM-4386] Examples and documentation for CalibrateTask were updated to modern stack conventions. [DM-4125]
 
A significant series of updates to the Hirata-Seljak-Mandelbaum shape measurement code (shapeHSM) were ported from Hyper Suprime-Cam. This included corrections to masking, scaling of errors, and better unit tests. [DM-3384]
 
Continued work on understanding the computational performance requirements for measuring galaxy shear. The significance of previous results has been lower than anticipated, so we concentrate on increasing the number of trials and improving the accuracy of error estimation. Following this work, a clear signal is seen and we now have a solid foundation for starting to draw conclusions from this work. [DM-3984, DM-3983]
 
Planned activities:
 
02C.04.00 - Data Release Production Management Engineering and Integration
 
Initial interviews will be carried out for the Postdoctoral Positions in Software. Continued documentation updates and refinements, in particular involving review by Lupton, will be incorporated to the long-term plan. We will be providing input and requests to the Data Access group regarding future design and development of the ‘Butler’ data loader.
 
02C.04.01 - Application Framework for Catalogs
 
Continued focus on merging work from HSC. In particular, recent work on HSC has concentrated on producing a diagnostic and QA measurement suite to verify their software releases. This will be ported to LSST to confirm that the processing of HSC data through the LSST stack is proceeding correctly.
 
02C.04.02 - Calibration Products Pipeline
 
Analysis of the data recorded at CTIO in November.
 
02C.04.03 - PSF Estimation
 
No work is planned for this WBS element in this month.
 
02C.04.04 - Image Coaddition Pipeline
 
No work is planned for this WBS element in this month.
 
02C.04.05 - Object Detection and Deblending
 
No work is planned for this WBS element in this month.
 
02C.04.06 - Object Characterization Pipeline
 
When the verification script described in 02C.04.01 becomes operational, it will be used to confirm the correct operation of the various measurement routines which have been ported from HSC.
 
Work will also proceed on updating the LSST measurement tasks to take advantage of the parallelization framework ported from HSC in November (and described above).

 


IPAC / California Institute of Technology

Current accomplishments:

02C.05.00  Science User Interface and Analysis Tools Management Engineering and Integration

·  Continue to work with IPAC IRSA group on collaboration in Firefly development, plan and schedule coordination.

·  Xiuqin, David, and Gregory will attend the DMLT meeting at Princeton.

·  Xiuqin vacation: 2 days

02C.05.01   Basic Archive Access Tools

·  Attended the bi-weekly discussion with SLAC, NCSA, SQUARE on authorization and authentication

·  Continued working on the conversion of Firefly client side software from GWT based code to React/FLUX based pure JavaScript code

o   Upgraded React to 0.14.3; 

o   finished converting some FITS visualizer functions to JS, grouping of images, zoom/scrolling separately or by group;

o   finished the banner and menu conversion to JS.

·  Continued updating the SUI/T design and design document

·  Provided a framework with grid image display feature (to display multiple images at the same time) for SQuaRE group. 

·  Fixed a memory leak bug in Firefly 

02C.05.02   Data Analysis and Visualization Tools

·  finished the histogram options in UI;

·  Provide support for Camera team to use Firefly to display the focal plane image, discussed using the function in Firefly to overlay mask image on primary image. 

Planned activities:

02C.05.00  Science User Interface Management Engineering and Integration


·  Continue to work with IPAC IRSA group on collaboration in Firefly development, plan and schedule coordination. Finish the move to github.
·  Finish the Winter 16 re-planting and the DLP 
·  Caltech closes from Dec. 25, 2015 - Jan 3, 2016
·  Xiuqin and  vacation: 4 days

 
02C.05.01   Basic Archive Access Tools
 

·  Continue to attend the bi-weekly discussion with SLAC, NCSA, SQUARE on authorization and authentication
·  Continue working on the conversion of Firefly client side software from GWT based code to React/FLUX based pure JavaScript code
·  Continue working on the SUI/T design and design document
·  Support SQuaRE work in visualization using Firefly

 
 02C.05.02   Data Analysis and Visualization Tools
 

·  Provide support for Camera team to use Firefly to display the focal plane image

 
 02C.05.05 User workspace

·  Continue the design of workspace.

 
 

 


SLAC / Stanford University

Current accomplishments:
02C.06.00 Science Data Archive and Application Services Management Engineering and Integration
 


·  Coordinated November Sprint for the Data Access Team
·  DMLT face to face meeting in Princeton
·  Organized weekly Qserv and Data Access meetings
·  Long-term planning optimizations
·  Small W16 replan
·  Continued search for candidate for the remaining open position
·  Actively working on XLDB2016 conference program at SLAC

 
02C.06.01.01 Catalogs, Alerts and Metadata
 

·   DM-3960: First optimizations of provenance querying
·   DM-3959: Revisit provenance sizing

 
02C.06.02.01 Data Access Client Framework
 

·   DM-4170: Butler: move configuration (.paf) file into repository

 
02C.06.02.03 Query Services
 

Release highlights:
·   Progress with Implementing Database & Table Mgmt
·   Improvements to query cancellation code
 
Code improvements:
·   DM-4138: Re-implement packed keys in CSS
·   DM-2622: Modify czar to support table deletion
·   DM-2624: Implement DROP table in watcher
·   DM-2625: Create service for managing watcher
·   DM-3945: Simplify task queuing / Runner code
·   DM-3563: Add unit tests to exercise new scheduler
·   DM-3564: Integrate Qserv code with cancellation-friendly xrdssi
·   DM-4395: Update cmsd configuration for multi-node tests
 
Bug fixes:
·   DM-4206: wmgr should delete database from inventory when dropping it
·   DM-4398: Fix regexp for gcc48
·   Continued work on data distribution prototype
·   DM-4231: Data Distribution prototype
·   DM-3569: Research/reading for PAST prototype C++
·   Continued work on secondary index:
·   DM-4183: Experiment with memcached for secondary index
·   DM-4184: Experiment with xrootd for secondary index

 
Planned activities:
 
02C.06.00 Science Data Archive and Application Services Management Engineering and Integration
 

·  Organize weekly Qserv and Data Access meetings
·  Search for candidate for the remaining open position

 
02C.06.01.01 Catalogs, Alerts and Metadata
 

·   Continue work on provenance design

 
02C.06.02.01 Data Access Client Framework
 

·   Continue work on Butler
·   Improve spatial image search

 
02C.06.02.02 Web Services
 

  
    
· Add unit test to Webserv
02C.06.02.03 Query Services
· Revisit shared scans design
· Complete work on database/table management
· Continue work on data distribution prototype
· Complete work on query cancellation
· Continue work on secondary index


NCSA / University of Illinois


Current accomplishments:
02C.07.00 Processing Control and Site Infrastructure Management
 
Design specification – In November NCSA spent considerable time working through the design specification of the Level 1 system. This included a deep dive into LDM-230 (Automated Operations of LSST DM) and documentation about the OCS-DM interfaces, functional components of the L1 system (e.g., EFD replication, calibration products production), and system responses to various error cases (e.g., backlog of Alert Production, dropped rafts, network outages). A significant result was that we proposed a simplification to the design by eliminating the network outage buffer and expanding the buffer in the LSST camera. We simultaneously worked on developing a more detailed WBS breakdown of processing middleware work needed for constructing and delivering this L1 system. We view this WBS as a backlog “to do” list, as it is currently not scheduled and or resource-loaded. In its present state, the backlog has over 300 items.
 
Operations Planning – Participation in the LOPT and TOWG continued in November. Accomplishments included drafting a detailed operations WBS for the data processing directorate. We also reviewed our detailed catalog of services to be provided by the Archive Center/DAC, and used it to provide an initial description and estimate of the effort needed during operations to deliver these services.
 
Joint Coordination Council – The JCC meeting with CC-IN2P3 in November was our biannual face-to-face meeting at NCSA. This meeting included an Extended JCC meeting with high energy physics facilities, including ANL and NERSC, with the goal of understanding the HPC environment that is available to people seeking to exploit the LSST stack at these centers. Coordination with CC-IN2P3 as a satellite computing facility included discussions of inter-site data exchange using object stores, coordination of data processing ahead of formal survey operations in the context of the MREFC construction project, and current explorations at each facility in storage, computing, and data format technologies. We identified technical and financial aspects to study jointly going forward, including data transport protocols (e.g., HTTP2, gridFTP), ability to implement levels of quality of service in accessing data for DRP and for end-user access, and measuring the impact on capacity and latency of compute nodes if using a networked file system vs. staging to local disk. Notes from both the JCC and Extended JCC meetings are available here: https://confluence.lsstcorp.org/display/JCC/Extended+JCC+meeting+--+2015-11-23 , and here: https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41789224 .
 
Sizing Model – In November we attended the week-long SC2015 conference in Austin. This involved discussions with vendors about technology trends and cost forecasts relevant to the technologies and timeline needed for LSST. The insight gained will be used when updating the technology and costing estimates in the Sizing Model for the February refresh.
 
FY16 Hardware Purchasing Plan – Approval of the FY16 hardware contract amendment by AURA was still pending in November. This amendment included the Annual Acquisition Strategy document describing general capabilities NCSA intends to acquire in FY16. In anticipation of the amendment’s acceptance, we began drafting the FY16 procurement plan that details the specific procurement activities, and timeline of these activities, we plan to make this fiscal year. We solicited quotes and configurations from multiple vendors across the full year of purchases. Additionally, we derived power requirements and rack layout, and provided a plan for LSST’s hardware footprint in the National Petascale Computing Facility over the lifetime of the survey.
 
Additional management activities – NCSA attended the LSST DMLT meeting at Princeton, where we discussed our current efforts and accomplishments, our short-term and long-term plans and the integration intra- and inter-subsystem, governance and organization of DM, processes for improving communication and productivity, and milestone linking between JIRA and PMCS.
 
Hiring – In November we made an offer to Mikolaj Kowalik for the position of Research Programmer. He accepted our offer and we hope to have him begin in February.
 
02C.07.01 Processing Control
 
Process Execution Framework – November work involved finishing the preliminary refactoring and expanding of the Supertask workflow in preparation for a demonstration at the DMLT meeting in Princeton, as well as preparing preliminary documentation. Following the DMLT meeting, we held an organizational meeting to layout and prioritize the work going forward with developing SuperTask.
 
Prototype DRP sequence with DECam data – In November we made improvements to the DECam ISR codes in the LSST stack, reduced a small set of DECam data, and began testing orchestration, workflows, and using the Nebula for processing:


·   Improved the overscan correction code to correct for the bias jump in raw images caused by the way in which the pixels are read out, and we demonstrated the discontinuity is now removed; see before and after images at https://jira.lsstcorp.org/browse/DM-4366 .
·   Modified the fringe subtraction code so that during runtime it is sensitive to the filters that are present.
·   Helped with testing and migrating DECam test data to git-lfs, which involved restructuring the dataset to reduce it from 1.2 GB to 120 Mb.
·   Started learning about the interface of tasks with the Butler, and the data storage and retrieval that is involved in ISR processing.
·   Processed a small set of DECam raw data through current ISR and calibration steps and discovered issues with the data that will need additional investigation, including glowing edges and poor astrometry.
·   Tested the Orca orchestration framework for launching single frame processing jobs using the LSST development cluster at NCSA.
·   Tested the Nebula cluster for development and processing, including creating a nebula instance with the stack, updating and installing more packages in the stack, constructing a data repository, and processing ISR with DECam raw images.

 
AP Simulator – We started writing up documentation on how the AP simulator currently works, including processes are started and how messages are sent through the Base and Archive DMCS. We also began a project to simulate realistic raw camera data. These data will be used for timing DMCS prototype codes and the building of camera images from raw pixel data. Two samples were built to represent a single LSST camera raft: one additively built and converted DECam data, and the other was additively built up from PhoSim data.
 
02C.07.02 Infrastructure Services
 
Identity and Access Management (IAM) – At the beginning of November the Identity Management group finished the draft LSST IAM Design Document (available here: https://confluence.lsstcorp.org/x/bKV9Ag ). This document describes LSST’s current and expected IAM needs and specifies technical recommendations for the LSST IAM system architecture, including system interface standards and system components, with functional descriptions and implementation recommendations. We then proceeded with setting up an initial implementation of the LSST IAM design. This technical demonstration is done in a VM testbed utilizing the Nebula cluster resources and includes SSH access (using NCSA Kerberos and LDAP), DAX, MariaDB, and WebDAV.
 
02C.07.03 Environment and Tools
 
No activities are planned for the W16 cycle.
 
02C.07.04 Site Infrastructure
 
Replacement development infrastructure (FY15) – As of November we received all equipment and licensing purchased under the FY15 hardware contract, and worked most of the month on installing and configuring the new hardware. We installed new Dell servers to host the new VMs, converted three physical systems used for system monitoring and configuration to VMs, deployed six new VMs as a Condor compute cluster, and migrated all VMs to the new vSphere infrastructure for central management of all VMs. We also installed base VM templates for Mac build slaves used for continuous integration, including OS X 10.8-10.11. Additionally, we completed installation of the UPS connections for the storage servers, which ensures data integrity and continuous operation of the development environment.
 
Decommission old infrastructure – Simultaneous with deploying new infrastructure, we also began decommissioning old infrastructure. Much of this involved shutting down physical machines that were moved to VM infrastructure (e.g., the old Condor servers).
 
General system administration – Other general system administrative activities in November included:

·   Replacing failed drive in storage server
·   Upgrading messaging system during monthly maintenance
·   Setting up VM for IAM technical demonstration testbed
·   Setting up six new accounts, including IAM group

Commissioning OpenStack for LSST – We created introductory guides for Nebula usage to enable LSST users to get started. The guides include: 1) a sample work session to start up an instance of a standard base operating system image and connect to the instance, and 2) sample work session to start an instance based on a previously prepared snapshot containing a preinstalled LSST software stack and using a data volume of pre-staged data. The guides are available here: https://confluence.lsstcorp.org/display/LDMDG/NCSA+Nebula+OpenStack+User+Guide
Long-Haul Network from Base Site to NCSA – We have been working on prototyping the layer 2 circuit LSST will have by building AL2S VLANs from Miami to NCSA, and as of November the circuit was built into NCSA. Additionally, we consulted with Ron Lambert on the Base Site networking architecture, including equipment procurement options and the South American RFP process, as well as rack layout options, sizing considerations, and impact on data center design.
 
Consulting on Chilean Data Center Contract – In November we reviewed design documents and corresponded with the design team regarding Data Center details and floor space.
 
Planned activities:
 
02C.07.00 Processing Control and Site Infrastructure Management
 
Design specification – We will continue our deep dive into the Level 1 system. We plan to finish the design specification of L1 to adequate depth for detailed planning. We will start by drafting a Concept of Operations of the L1 system, as well as revisit the design diagrams and documentation in Confluence ( https://confluence.lsstcorp.org/x/B4B3 ).
 
Operations Planning – In December, we will work on providing descriptions of the operations WBS elements and further refine our estimated FTE counts. We will develop material for the system diagrams that are meant to show a 24-hour cycle of activities at the NCSA Archive Center. Finally, we will host Beth Willman at NCSA for a two-day meeting to discuss operations planning.
 
Sizing Model – We will begin the technology and costing refresh of the Sizing Model.
 
FY16 Hardware Purchasing Plan - December is the 60-day turnaround for the hardware procurement contract amendment to be approved. Pending this, we plan to being procuring specified capabilities in December. We will continue to research vendors and costing so we are prepared to make procurements immediately following the contract approval.
 
02C.07.01 Processing Control
 
Process Execution Framework – In December we will clean up the existing SuperTask codes in the development branch, add comments and technical documentation, and migrate the package in the main LSST stack repository so that developers can begin testing it.
 
Prototype DRP sequence with DECam data – In December we will work on expanding the LSST stack to implement ISR corrections not yet available for processing raw DECam data, e.g., crosstalk, edge-bleed, and illumination correction. We will also work on coadding images and identifying issues with the current implemented ISR corrections that could severely impact the quality of post-ISR processing.
 
AP Simulator – In December we will utilize the simulated images mentioned above to test the timing of building them from pixel data and propagating them through the DMCS. To get started, we will compare the timing of different compression techniques.
 
Network Emulator for AP – In December we will start working on deploying the Network Emulator purchased in the FY15 hardware procurement, working towards the ability to simulate data movement between two sites.
 
OCS Software Integration – We anticipate a new version of LSE-70 (Communication Protocol Interface) from the OCS group in December, and we will begin to evaluate these in the context of Base DMCS receiving and responding to OCS messages.
 
02C.07.02 Infrastructure Services
 
Identity and Access Management (IAM) – Preparing the LSST IAM technical demonstration will continue in December. Additionally, we will consider the process for granting data access right to LSST users, e.g., automatically based on institutional attributes (InCommon authentication) vs. by review for named individuals, etc.
 
02C.07.03 Environment and Tools
 
No activities are planned for the W16 cycle.
 
02C.07.04 Site Infrastructure
 
Replacement development infrastructure (FY15) – We have most of the NCSA LSST servers under centralized configuration management using Puppet, and in December we will investigate migrating the Mac VMs to Puppet as well. We will also work with the SQuaRE team to understand their needs of the Mac VMs.
 
Commissioning OpenStack for LSST – We will expand on the Nebula guides for LSST users. We will also begin investigating the use of Nebula as a HTCondor pool for batch data processing with the LSST orchestration framework (orca).
 
Commissioning new infrastructure (GPFS, OpenStack) – As stated above, in the event the hardware contract amendment is finalized, we will begin purchasing FY16 equipment.
 
Long-Haul Network from Base Site to NCSA – In December we will continue consulting with Ron Lambert on the Base Site networking design, update the network end-to-end test following the outcome of the networking group telecon in December, and pending the approval of a new baseline LSE-78 (network design document), propose new updates to LSE-78 with accurate information about the LHN and the Commissioning Cluster architecture.
 
Consulting on Chilean Data Center Contract – We will participate in the Base Facility preliminary design review and any actions that result from the review.

 


AURA/LSST and AURA/NOAO


Current accomplishments:
02C.08.00 International Communications and Base Site Management Engineering and Integration
 
Participated in the ITWG group at the request of Victor to discover if the institutions may be integrated in some fashion for common work areas.
 
02C.08.01 Base Center
 
Participated in the Base Design review. Between NCSA, Jeff Barr and the Andes Architects we went though many iterations of the floor plan size for the Data Centre. The original problem was that not all machines counts were being considered and we went from ~90 racks to ~120 racks so the area was increased by the Architects to accommodate those machines. We also increased the area of each rack from 684 sq.in to 1416 sq.in so the racks are now 29.5” wide x 48” deep x 48U from the original 19” x 36” x 42U. It was felt that the racks in the DC should be uniform throughout the room. The room height and ceiling plenum were also increased and added.
 
02C.08.03 Long-Haul Networks
 
This month I attended the SC2015 in Austin to meet and discuss Base Data Center networking with Paul and Jason from NCSA. It was very useful to touch base with the NCSA personnel in terms of fostering working relationships.
 
Following on from Austin I visited Tucson to talk to Iain and Shahram regarding the LSST current networking needs for the group now in La Serena. They will shortly send down a router/firewall to form a separate subnet for LSST. We also subdivided the remaining 32,000 addresses available in the 139.229.0.0/16 Class B space for LSST on the summit and at the base. The whole issue of IP addressing is somewhat undecided from my perspective whether LSST will utilize our current space, use private addressing or IPv6 or some combination of all those protocols. I don’t see the value in NAT as it breaks the fundamental policy of the internet and we do have adequate address space either Private for internal, IPv4 or IPv6.
 
Participated in the DWDM vendor selection with Reuna and Fernando Liello. This was the second round and we now have all the vendors on the same page regarding the technical solution. The next round(s) will primarily be to haggle over price but we already have 4 of the 8 under the $1.4M budget; Fernando feels this can be improved along with extra options such as 2nd Lambda for AURA summit to base and upgrade of Santiago ring to support 100G alien Lambdas for LSST and Bella. Third round is set to be 18 January 2016.
 
Visited with Jacques to discuss the Telescope network at the Summit. We reviewed the summit facility document that I had presented. Jacques left me work to do on pricing multi-strand OM3/4 fiber cable and diagramming the summit network. Both are completed.
 
Planned Activities.
 
02C.08.00 International Communications and Base Site Management, Engineering, and Integration
 


·  Work on requirements for AURA and LSST regarding the ITWG

 
02C.08.01 Base Center
 

·  Continue with Base design meetings
·  Work on Gantt chart for Base and Summit infrastructure facilities
·  Revise power requirements for the Base Data Center

 
02C.08.03 Long-Haul Networks
 

·  Continue LHN contracts and design
·  Complete Gantt charts for the LHN to NCSA
·  Complete DWDM vendor evaluations prior to January

Back to top