1. Project Status
  2. Current Photos
  3. Risk Management
  4. Detailed Project Progress and Status
    1. LSST Program Office
    2. University of Washington
    3. Princeton University and University of California, Davis
    4. IPAC / California Institute of Technology
    5. SLAC / Stanford University
    6. NCSA / University of Illinois
    7. Current accomplishments:
    8. NOAO
    9. Current accomplishments:

Data Management Monthly Report

September 2015
 
Project Status


·  The Summer 2015 release was completed , including final metrics analysis runs, release notes, and other documentation. Some late in the cycle features include:
o   Enabled Eigen-based astrometry, so that Astrometry.net is no longer the default astrometric solution, but it is still available as an option.

o   A general purpose n-way spatial matching tool was produced. This was developed primarily to support measurement of Key Performance Metrics, but is generally applicable and usable elsewehere.

·  The Winter 2016 release was started. Some highlight features include:
o   Completion of the merger of the Hyper Suprime-Cam codebase to LSST

o  A reworking of the top-level LSST pipeline tasks to better reflect the new functionality available after the HSC merger
o   Initial work on major upgrades to the deblender, starting with an overhaul of the Footprint system

o   First version of chromaticity support in the stack

o  Continued work on galaxy fitting
o   A new Firefly core system using React and FLUX framework. Conversion of Firefly client side software from GWT based code to React/FLUX based pure JavaScript code

o  SQLite backend to db module
o   Support for registry-free repository in butler
o  Improved spatial image search and provenance
·   The FY16 Infrastructure Acquisition Strategy document was prepared and will be reviewed on October 8. Once approved by the LSST Project Office, a change request will be submitted to adjust the baseline acquisition schedule more front-loaded for the Development and Integration clusters.
·   Completed assessment of the cost of keeping more than two data releases on spinning disk. This storage would increase the size of the coadd and calibration image caches and the L2 database capacity need, and using LDM-141 (noting that the sizing model costing refresh in August did not fix all inconsistencies attributed to out-of-date technologies) we estimated the cost increase is minimal.
·  We continued work on the Data Management Development Roadmap (LDM-240), improving Milestone and Key Performance Metric descriptions and analyzing resource loading.
·  The MOA (Networks/PIs) has been signed by all parties and is complete.
·  Continued execution of the RFP for the Long-Haul Network DWDM end node equipment. We have nine proposals and interviews to read and conduct.
·  The DM Project Scientist continued the verification and understanding of NEO detection efficiency and effectiveness.
·  The DM Project Scientist participated in the PST face-to-face meeting in Tucson. He delivered and presented a written report of an assessment of the expected science effects of having a heterogeneous focal plane (from the DM point of view).
·  The DM Project Engineer and other DM team members planned the DM Boot Camp to be conducted simultaneously at LSST Tucson, Princeton, and UW. The Boot Camp will train the new hires in developing using the DM processes, tools, and documentation.
·  Recruiting and hiring activities continued across all DM institutions. Twenty-nine positions have been filled to date since the MREFC award, while 8 positions are currently open.   One offer was made and accepted. Several new hires started in DM institutions and one new hire started in Tucson.
·  The DM SQuaRE team started using the OpenStack-based cloud development environment at NCSA and provided feedback on the implementation, missing features, etc.
·  The DM SQuaRE team started verification activities with a focus on processing six datasets, mostly DECAM, with the stack.  Two datasets have been processed through the first stage of the stack, processCcd, one of them using the legacy orchestration software (orca) in the absence of any alternatives. Skymaps have also been produced for at least one of these datasets.

 

Back to top



Current Photos
 
No current photos to submit (Suzanne Jacoby has pictures from DM Boot Camp for next month report).
 
 
 

Back to top



Risk Management
 
The DM Risk Register was reviewed in the monthly process. Based on the successful development and ongoing implementation of the Information Security Program, reductions in probability/risk exposure were made in several cyber-security related risks.

 

Back to top



Detailed Project Progress and Status


LSST Program Office

DM Project Management and Control

Current accomplishments:

The DM Project Manager:


·  Briefed the AMCL on DM status at the September 3 meeting at SLAC
·  Completed developing the FY16 budget and started preparing the FY16 sub award contract amendments.
·  Continued work with NCSA on FY15 infrastructure procurements and FY16 Infrastructure Acquisition Strategy.
·  Completed development of the Brazil MOA (Networks/PIs). The MOA is now executed.
·  Continued development of the contracts for the Chilean National and Mountain – Base fiber optic networks and corresponding updates to schedules in PMCS.
·  Created a draft ENews article regarding Networks.
·  Prepared presentations for the DM Boot Camp to be held simultaneously at the LSST Project Office, Princeton, and UW on October 5 – 7.
·  Participated in Export Controls training and program development.
·  Participated in Intra-Project Communications improvement activities.
·  Recruiting and hiring activities continued across all DM institutions. Twenty-eight positions have been filled to date since the MREFC award, while nine positions are currently open.

 
Planned activities:

The DM Project Manager will:


·   Participate in the DM Boot Camp.
·   Continue recruiting and hiring, open new SQuaRE scientist position.
·   Continue development of the contracts for the Chilean National and Mountain – Base fiber optic networks and corresponding updates to schedules in PMCS.
·   Complete work with NCSA on FY15 infrastructure procurements and FY16 Infrastructure Acquisition Strategy.
·   Complete ENews article regarding Networks.
·   Continue development of FY16 contract amendments.
·   Plan next face to face DM Leadership Team Meeting at Princeton November 16 – 19.

 

DM Science

Current accomplishments:
September activities continued to be focused on the study of the efficiency of
LSST as a detector of NEOs, onboarding of new staff, defining DM project science
development priorities for Winter 2016, as well as participation in a series of
meetings.
 
Specific work includes:
 
Finalization of the first phase of the NEO detection efficiency study and the presentation of the work to NASA. At the invitation of John Grunsfeld, the Associate Administrator for the Science Mission Directorate at NASA, The DM Project Scientist presented the results of this work at the NASA headquarters, together with Steve Kahn and Zeljko Ivezic (representing LSST) and Matt Mountain and Heidi  Hammel (from AURA). Our analysis shows that within reasonable uncertainties regarding “false positives” that may arise in our image differencing algorithms, existing experience with DECam makes it very likely that LSST will detect  ~70% of NEOs down to H=22. With some optimization of the cadence strategy and an extension of the survey to 12 years our preliminary analysis shows we can can increase the fraction to 85% or higher. It is noteworthy that this analysis and report included a cross-subsystem team working together, including the Simulations team lead by Andy Connolly that delivered the cadence explorations, Lynne Jones and Zeljko Ivezic (from Project Science), as well as Data Management Project Science (Colin Slater, with myself coordinating the overall effort).
 
Colin Slater has started at University of Washington (50% FTE). He will support DM Project Science (DMPS) work over the next three years. He has been on-boarded and has started initial work on analysis of the quality of image differencing prototype codes in support of the NEO detection study.  We are pursuing one more hire to further staff up the DMPS area.
 
The DM Project Scientist constructed and delivered to the DM Project Manager and Engineer a list of key short-term development priorities from the point of view of DM Project Science. This will be folded into the plans for Winter 2016 development.
 
With Robert Lupton and K-T Lim, the DM Project Scientist met (via videocon) with the leadership of  EUCLID NIR data processing team to explore the possibilities for technical  coordination or collaboration on code development in certain areas. At this  point, the discussions are tentative and highly exploratory (though promising).
 
The DM Project Scientist participated in the PST face-to-face meeting in Tucson. He delivered and presented a written report of an assessment of the expected science effects of having a heterogeneous focal plane (from the DM point of view). 
 
The DM Project Scientist gave a talk on LSST at the "Frontiers in Computing and Data Science” at MSU.
 
 
Planned activities:
 
In October, the DM Project Scientist expects to remain focused on the NEO studies (especially any activities resulting from the NASA meeting),assist with the finalization  of long-term DM development planning activities, and discuss LSST at a series of conferences and workshops (most importantly, Astroinformatics 2015 and ADASS XXV).
 
DM System Engineering

Current accomplishments:

Accomplishments by the DM System Architect and Deputy System Architect include:


·   Prepared for and attended AMCL Board meeting
·   Met with DESC spokesperson to synchronize roadmaps
·   Completed planning and reviewed some presentations for DM Boot Camp
·   Took export control training
·   Verified OCS/Camera integration milestones
·   Redefined release process and versioning
·   Defined planning process leading up to Nov DMLT in-person meeting
·   Did processing middleware and Data Butler design work
·   Met with Intel performance experts
·   Assisted with 2016 hardware procurement planning
·   Assisted with definition of DM operational systems
·   Planned and coordinated DM Boot Camp
·   Continued work to finalize DM Long Range Plan, address open issues prior to DM Leadership Team Meeting November 16 – 19 at Princeton
·   Continued leading DM System Architecture and DM representation on LSST System Engineering Team
·   Helped port Qserv to OS X (DM-3584 / DM-3803). Qserv now builds with clang on OS X with very few compiler warnings.
·   Assisted with release engineering for v11.0 (DM-3815)
·   The boost and Eigen third party packages were upgraded to current versions. The Eigen update highlighted some algorithmic instabilities in earlier versions. In particular Mac and Linux discrepancies from fitting algorithms were removed when Eigen was updated. (DM-3699)
·   Began comparing the Data Management System Requirements (LSE-61) with LSST system requirements, ICDs and DM design documents in order to add requirements traceability.
·   Prepared a presentation for the DM Boot Camp

 

The DM System Interfaces Scientist accomplished:


·   Vacation 9/1-9/8, 9/14, 9/23 (8 of 22 working days)
·   Worked with Brian Selvy to revise and agree on an achievable scope for LCR-323 (handoff milestones for system integration) for agreement at the September CCB meeting.  (Final decision was ultimately deferred to October CCB because of Camera decision-making process.)
·   Separated out a further chunk of work for a second phase, mainly related to the incorporation of the Camera DAQ system into the flow of integration activities already defined.  Draft document developed with Mike Huffer and Brian Selvy:
https://confluence.lsstcorp.org/display/SYSENG/DAQ+integration+steps
·   Multiple rounds of work with Matias Carrasco Kind to refine the "SuperTask" project design.  Resulted in class diagrams and prototype code developed by MCK.
·   Trip to Tucson to work with the SQuaRE group to explain the SuperTask design and ensure that it is useful in the SQuASH-harness context for data quality analysis jobs.
·   Stepped up efforts to ensure that the Camera visualization effort progresses, concentrating on improving communication and on encouraging camera management to take closer interest in the project's progress.
·   Ongoing work with John Rector and others on refining the SUI / Workspace architecture.
·   In the context of LCR-385, reviewed LSE-78 (network design) and engaged Don Petravick, Dominque Boutigny, Chuck Gessner, and others with some questions the review raised.  Prepared list of issues for Ron Lambert (who was on vacation at the time).
·   Held a teleconference review of LSE-70 and LSE-209 within DM and provided feedback to Brian Selvy about the usability and readiness of the documents.

 
Planned activities:

The DM System Architect and Deputy System Architect will:


·  Continue work to finalize DM Long Range Plan, address open issues prior to DM Leadership Team Meeting November 16 – 19 at Princeton
·  Continue leading DM System Architecture and DM representation on LSST
·  Conduct/Attend DM Boot Camp
·  More Data Butler design work with Nate Pease
·  Complete 2016 hardware procurement planning
·  Further refine definition of DM operational systems
·  Set DMLT meeting agenda
·  Prepare poster for and attend the ADASS conference.
·  Continue to work on DM requirements tracing.

  
    
                
 
 
The DM System Interfaces Scientist will:
· Final decision on scope of LCR-323 with Brian Selvy
· Continue work with John Rector and others on refining the SUI / Workspace architecture.
· Continue work on “supertask”
DM Science Quality and Reliability Engineering (SQuaRE)
Current Accomplishments:
02C.01.02 SDQA 
· Summer 2015 release
The “Summer 2015” version of the stack (aka v11.0) was released. Extensive release notes can be found at http://ls.st/0qg
V11.0 is the first release accompanied by a characterization summary ( http://ls.st/wek ). These reports will become more substantial as construction progresses and SQuaRE is able to measure an increasing number of Key Performance Metrics.
In technical terms, this is the first release with CentOS 7 as the factory platform and gcc 4.8 as the minimum version. It was successfully tested on a variety of platforms including CentOS 6 (the old factory), Ubuntu, OSX and (by request) OpenSUSE. During Summer 2015, CI was done simultaneously on CentOS 6 and 7; as a result, we noted a lot fewer portability issues during release testing, as many were uncovered and fixed as part of the normal development process. We hope to further increase the number of CI platforms as our capacity increases in FY16.
An extension of the new CI system to perform a high frequency of automated builds with the user-facing process (newinstall.sh) proved extremely useful in uncovering a number of intermittent build errors that could be addressed before the release shipped. We have already began to reap the benefits of our automated cloud-based CI infrastructure, with many improvements still to come.
SQuaRE’s release process also improved in the light of lessons learned from v10.1 and as a result the technical part of the process was significantly smoother (and shorter) than in previous releases, with minimal impact on developers who were not directly involved in fixing bugs blocking the release. The next release will focus on improvements in documentation preparation, as this is now the most time consuming step. We are improving our code, systems and process at each release not only to reduce the effort involved, but also to ensure that we can sustain a high pace of high-quality “public” releases around Integration & Testing and Commissioning.
· Science Verification Activities
Verification activities started in earnest with a focus on processing six datasets, mostly DECAM, with the stack. A number of problems caused a slow start to the work: the stack has a steep learning curve for new users/teams, and a number of technical difficulties were encountered in particular with obs_decam. The suggestion has been made that SQuaRE “adopts” the obs_* packages, and we are evaluating how we can support this.
Two datasets have been processed through the first stage of the stack, processCcd, one of them using the legacy orchestration software (orca) in the absence of any alternatives. Skymaps have also been produced for at least one of these datasets. Hsin-Fang (NCSA) has been working on obs_decam ISR and getting the calibration products/registry to work.
While these activities are of intrinsic interest for our QA activities characterizing the algorithmic perfomance of the stack, they also provide a valuable opportunity to capture the usability of the stack as an astronomer-oriented tool. David Nidever, Colin Slater, Jonathan Sick and Angelo Fausti seeded an initial list of deficiencies ( http://ls.st/yqj ) based on their initial experience, and it is expected that this list will continue to evolve. We will fold items on that list actionable by SQuaRE into the planning for future cycles (where they have not been identified and planned for already).
Minutes for the Verification weekly telecons have be found at http://ls.st/f43
· Other Technical Activities
We started to investigate GitLFS as a solution to storing data repositories of CI/QA interest (such as afwdata) in git. Work commenced to evaluate the technology and, if promising, to deploy a test service and bring it to RFC.
Following the adoption of Discourse (RFC-85) at the end of August as a web forum platform for DM, work was done to support its deployment on community.lsst.org (or as it is increasingly referred to, c.l.o).
Following discussion at the Bremerton meeting, work started on a sphinx-based documentation infrastructure for the DM codebase.
· Non-Technical Activities
Frossie Economou visited the team at SLAC and gathered input relating to SQuaRE developer infrastructure activities.
Planned Activities:
02C.01.02 Science Quality and Reliability Engineering
· Verification dataset processing
· GitLFS test deployment & RFC
· NCSA Nebula Openstack cluster evaluation (cont)
· Documentation support for Plan-for-Planning process
· DM Boot Camp
DM Applications, Middleware, and Infrastructure
Current accomplishments:
· Refer to by institution reports below
Planned activities:
· Refer to by institution reports below


University of Washington

Current accomplishments:

02C.03.00 -- Alert Production Management Engineering and Integration

Russell Owen (RO), Yusra AlSayyad (YA) and Simon Krughoff (SK) worked with new hires Ian Sullivan (IS) and David Reiss (DR) to come up to speed on LSST developer related topics.  RO, YA, and SK prepared materials for the DM bootcamp happening the first week of August.  SK worked with K-T Lim and the other T/CAMs on continued planning activities.

02C.03.01 -- Single Frame Processing Pipelines

The upgrade in Eigen also caused test failures in testProcessCcd.py.  RO fixed the tests (DM-3800).  This also fixed a long standing issue that some measurements were producing different numbers on OSX vs Linux.

02C.03.05 -- Application Framework for Exposures

As part of her work to get the validation datasets up and running, YA fixed tests in obs_decam so that it could more easily be distributed with the stack.  As part of a request to increase the fidelity of test data in the obs_test package, RO noticed and fixed issues with how the test data were being assembled (DM-3792).  RO also worked on various pieces of left over technical debt (DM-3698, DM-3691, DM-3651, DM-3670).  RO made some changes to improve how coadded PSFs are built (DM-3387).

02C.03.08 -- Astrometric Calibration Pipeline

RO fixed a variety of bugs exposed in meas_astrom by upgrading Eigen (DM-3824).  RO also made the new astrometry task the default task (DM-2792).  This means that Astrometry.net is no longer the default astrometric solution, but it is still available as an option.

Planned activities:

02C.03.00 -- Alert Production Management, Engineering and Integration

The first week will be taken up by the DM bootcamp.  Nate Pease will join the UW group in person and Russell Owen will be traveling to Tucson to assist with the bootcamp there.  SK and YA will be presenting at the UW bootcamp.  We will also welcome our last new hire, John Parejko.

02C.03.04 -- Image Differencing Pipeline

IA and DR will continue work on a DCR overview document.  YA intends to have a new version of the image differencing task running.

02C.03.05 -- Application Framework for Exposures

RO will be working to clean up the final pieces left over from bringing aperture corrections over from HSC.

 

 


Princeton University and University of California, Davis
This report covers work carried out in FM12 of FY15 in the Data Release Production group.
 
Current accomplishments:
 
02C.04.00 - Data Release Production Management Engineering and Integration
 
By request from Kantor and Long, the Winter 2016 plan was submitted on 15 September. This outlines the work we expect to perform through the next six months. Highlights include:


·  Completion of the merger of the Hyper Suprime-Cam codebase to LSST;
·  A reworking of the top-level LSST pipeline tasks to better reflet the new functionality available after the HSC merger;
·  Initial work on major upgrades to the deblender, starting with an overhaul of the Footprint system;
·  First version of chromaticity support in the stack;
·  Continued work on galaxy fitting, starting with research into fundamental paramters early in the cycle and transitioning to code and algorithmic improvements in later months. We expect to continue refining the schedule for this work with input from the DM Project Scientist (Juric) and Engineer (Lim). As part of the ongoing planning process, Bosch wrote a discussion document and plan de- scribing the current design for 02C.04.06 (the Object Characterization Pipeline). This document builds upon previous design documentation, notably LDM-151, to represent current algorithmic thinking in a way that will motivate the reworked software roadmap (LDM-240).
·  Substantial development activity across 02C.04 through this month focused on developing the measurement routines for the Key Performance Metrics to accompany the S15 release of the stack. This work is reported here because it cuts across the leaf elements of this WBS. Measurements of photometric repeatability (procCalRep, PA1gri, PA1uzy), relative astrometric performance (AM1, AM2), residual PSF ellipticity correlations (TE1, TE2) and the total DRP computational budget were provided, in every case coming below the target specified for FY15 (often substantially so). Measurements of AM3 were not available due to a lack of appropriate test data.

 
Hiring
 
Bosch, Lupton and Swinbank interviewed four applicants for the position of Scientific Software Developer on 3 and 4 September. An informal offer has been made to and accepted by one of the candidates; we are still waiting for the University HR department to process the paperwork.
We also interviewed one candidate for a postdoctoral position. This application is still under review pending a decision on work visa related issues.
The September issue of the AAS Job Register contains an advertisement for "postdoctoral positions in software", which will include work on LSST. We expect to begin reviewing candidates for this position in November.
 
02C.04.01 - Application Framework for Catalogs
 
The integration test currently run as part of the standard LSST build and test procedure performs some measurements on SDSS data and compares the results to a set of fiducial reference data. Inter-platform variability in numerical libraries renders this procedure error prone and unreliable. The test suite was therefore reworked to use appropriate per-platform reference data and to eliminate the use of the (slow and occasionally confusing) numdiff tool when performing comparisons. [DM-3688]
Assorted problems with building the PSFEx package as part of the stack were resolved. [DM- 3667, DM-3749]
A general purpose n-way spatial matching tool was produced. This was developed primarily to support measurement of Key Performance Metrics, but is generally applicable and usable elsewhere. [DM-3490]
An error with handling world coordinate system (WCS) headers in FITS files was corrected, which resolved an issue preventing loading DECam data into the LSST stack. [DM-3196]
Convenience routines to enable the creation and display of full focal plane mosaics were ported from Hyper Suprime-Cam. [DM-2437]
 
02C.04.02 - Calibration Products Pipeline
 
No work was carried out in this WBS element in this month.
 
02C.04.03 - PSF Estimation
 
No work was carried out in this WBS element in this month.
 
02C.04.04 - Image Coaddition Pipeline
 
No work was carried out in this WBS element in this month.
 
02C.04.05 - Object Detection and Deblending
 
No work was carried out in this WBS element in this month.
 
02C.04.06 - Object Characterization Pipeline
 
A number of updates to the star selection algorithms were ported from HSC. [DM-3678]. Work continued on exhaustively comparing side-by-side processing of the same data in the HSC and LSST environments, identifying and accounting for differences and, where appropriate, filing tickets to merge the appropriate changes. This is an essential and ongoing process to guide the continuing HSC port. [DM-3942]
Tests to determine the size, in pixels, of the region needed for fitting without unduly biasing galaxy shears were completed and the results written up. This is fundamental piece of research to inform future development, although the conclusion is less definitive than expected: it may be necessary to perform further tests on this in future. [DM-1135]
 
Planned activities:
 
02C.04.00 - Data Release Production Management Engineering and Integration
 
We expect the ongoing long-term planning project, and in particular updates to key project doc- umentation, to continue to require most of the time available from Bosch and Swinbank through October and until the DMLT meeting in November. This work will concentrate on developing design documents akin to the one already produced for the other DRP WBS elements, mapping those to the JIRA-DLP plan, and resource loading.
Many staff will be involved, as lecturers, tutors and/or attendees, at the the DM Stack "Boot Camp", which will take place in early October. Time has been allocated both for preparation by lecturers and tutors and for attendance.
Due to an HSC project meeting in Japan in mid-October, many staff will be away for much of October resulting in a reduced velocity.
 
02C.04.01 - Application Framework for Catalogs
 
Continued focus on merging work from HSC with particular emphasis on the parallelization middleware.
 
02C.04.02 - Calibration Products Pipeline
 
Fisher-Levine and Lupton will prepare to deploy the collimated beam projector at CTIO, an activity which is planned for November.
 
02C.04.03 - PSF Estimation
 
No work is planned for this WBS element in this month.
 
02C.04.04 - Image Coaddition Pipeline
 
No work is planned for this WBS element in this month.
 
02C.04.05 - Object Detection and Deblending
 
No work is planned for this WBS element in this month.
 
02C.04.06 - Object Characterization Pipeline
 
Continued focus on merging work from HSC with a particular focus on comparing the HSC and LSST pipelines for consistency, identifying and fixing points where they diverge.

 


IPAC / California Institute of Technology

Current accomplishments:

02C.05.00  Science User Interface and Analysis Tools Management Engineering and Integration

·   Xiuqin vacation: 5 days 

·   Continued working with IPAC IRSA group on collaboration in Firefly development, plan and schedule coordination, common system for issue tracking.

·   Finished Winter 16 resource loading

·   David Ciardi started working on the DAC definition, produced a set of questions.

02C.05.01   Basic Archive Access Tools

·   Continue the discussion of data access APIs with SLAC group if needed

·   Started the bi-weekly discussion with SLAC, NCSA, SQUARE on authorization and authentication. NCSA is leading the effort and the meeting minutes are posted at  https://confluence.lsstcorp.org/display/LAAIM/LSST+Authentication%2C+Authorization%2C+and+Identity+Management+Home

·   Resolved issues found in the DM end-to-end exercise.

·   Produced a draft design of the new Firefly core system using React and FLUX framework for discussion within the team next month.

·   Started working on the SUI/T design and design document. 

02C.05.02   Data Analysis and Visualization Tools

·   Finished the first version of the feature to overlay mask on the primary LSST image to support the needs for pipeline image visualizaiton.

·   Finished the feature to expose the function of region overlay on image through JavaScript API to support the needs of Camera team at UIUC.

·   Bug fixed in Firefly for coordinate grid overlay on images

·   Performance improvement made on FITS file read and visualization.

02C.05.03 Alert/Notification Toolkit

02C.05.05 User workspace

·  Started the preliminary design for workspace

Planned activities:

02C.05.00  Science User Interface Management Engineering and Integration


·  Continue to work with IPAC IRSA group on collaboration in Firefly development, plan and schedule coordination, common system for issue tracking.
·  Second round update in Jira project DLP, refine milestones,  KPM and meta-epics.
·  Attend the DM boot camp.
·  Xiuqin, David, and Gregory will travel to Tucson to meet with LSST deputy director.
·  Xiuqin and David will attend the ADASS meeting in Sydney, Australia.
·  Xiuqin vacation: 5 days

 
02C.05.01   Basic Archive Access Tools

·  Continue the discussion of data access APIs with SLAC group if needed
·  Continue to attend the bi-weekly discussion with SLAC, NCSA, SQUARE on authorization and authentication
·  Intensive design discussion (5 days, 3-5 hours per day) of the new Firefly core system using React and FLUX framework.
·  Start working on the conversion of Firefly client side software from GWT based code to React/FLUX based pure JavaScript code
·  JavaScript development environment setup, eslint, unit test, ect.
·  Continue working on the SUI/T design and design document

 
 
 02C.05.02   Data Analysis and Visualization Tools

·  Refactor some of the Firefly server side code

 
 02C.05.03 Alert/Notification Toolkit
 
 02C.05.05 User workspace

·  Continue the design of workspace.

 
 

 


SLAC / Stanford University

Current accomplishments:
02C.06.00 Science Data Archive and Application Services Management Engineering and Integration


·  Coordinated September Sprint for the Data Access Team
·  Finalized budget for the team for FY16
·  Input for / reviewing documents for the subcontract with IN2P3 for Fabrice
·  Organized weekly Qserv and Data Access meetings, participated in Authorization & Authentication discussions
·  Continued search for candidate for the remaining open position
·  Defined and modeled detailed Qserv KPIs
·  Planning XLDBGov2016 in D.C. and XLDB2016 at SLAC, started work on grant with NSF to support XLDB work at SLAC
·  Attended @Scale conference to follow up on latest trends in Big Data
·  Discussions with Stanford CS department about possible SLAC/Stanford collaboration

 
02C.06.01.01 Catalogs, Alerts and Metadata

·   N/A

 
02C.06.02.01 Data Access Client Framework

·   Reworked db module, which is relying on SqlAlchemy
·   Added support for type aliases

 
02C.06.02.02 Web Services

 
  
 
 
 
 
 
 
 
  
· Migrated Webserv to the new db module
02C.06.02.03 Query Services
· Release highlights:
o Simplified executor
o Added support for clang / Mac OS X
o Switched to SQLAlchemy based db module
o Reduced/isolated dependency on mysql
o Added support for IN queries
o Integrated Qserv with pipelines and SUI
o Ported to C++11 continued (nullptrs)
· Code improvements:
o DM-3562: Refactor executor code
o DM-2733: Generalize / Simplify Facade
o DM-3805: uint type is non-standard
o DM-3663: Add support for clang and OS X to qserv scons
o DM-3584: Port Qserv to OS X/Clang
o DM-3803: Fix Qserv compiler warnings with clang
o DM-3915: Unused local typedef warning in Qserv
o DM-3779: clean up gcc and eclipse code analyzer warns
o DM-3707: qserv scons - do not copy files to variant_dir
· New features:
o DM-3199: Standardize Qserv install procedure: step 1 build docker container for master/worker instance and development version
· Bug fixes:
o DM-3656: Data loader doesn't work for match tables
o DM-2887: Fix broken IN - it now takes first element only
o DM-3772: Fix compiler detection for non-default gcc/g++ compiler
o DM-3779: Fix compiler detection in Qserv for non-gcc compiler
o DM-3686: Fix PATH and compiler version detection in qserv scons
· Documentation improvements:
o DM-3922: Update multi-node setup documentation
Planned activities:
02C.06.00 Science Data Archive and Application Services Management Engineering and Integration
· Organize weekly Qserv and Data Access meetings
· Search for candidate for the remaining open position
· Resource loading for long term planning and start updating Data Access design documents and connect them with long term planning epics
· Define and model detailed image access KPIs
02C.06.01.01 Catalogs, Alerts and Metadata
· Improve provenance design
02C.06.01.02 Image and File Archive
· N/A
02C.06.02.01 Data Access Client Framework
· Add SQLite backend to db module
· Add support for registry-free repository in butler
· Improve spatial image search
02C.06.02.02 Web Services
· Add unit test to Webserv
02C.06.02.03 Query Services
· Improve xrdssi API
· Simplify interactions with xrdoss
· Replace zookeeper CSS with mysql
· replace qservAdmin.py with cssAccess
· modify CSS to support table and db deletion
· reimplement watcher
· modify czar to use per query CSS metadata
· handle “where objects between”
· data distribution prototyping
· simplify taks queuing / runner code
· integrate qserv code with cancellation friendly xrdssi
02C.06.02.04 Image Services
· Make location of images more flexible
02C.06.02.05 Catalog Services
· N/A


NCSA / University of Illinois


Current accomplishments:
02C.07.00 Processing Control and Site Infrastructure Management
 
Design Specification – The major focus of the NCSA team was breaking down the design specification of the “to-be” facilities. We are working to understand the functionalities to be built and to craft a fully detailed construction plan. We are documenting specifications, collecting requirements, and drafting functional breakdown diagrams on the LSST wiki ( https://confluence.lsstcorp.org/display/~petravick/Breakdown+of+the+to-be+facility ). In September we added the EFD, Observatory Operations server, and Chilean and US DAC, to the design. We investigated the data products flow and life cycles to understand file system and image cache needs. We also looked at the outputs of alert production and interactions of the L1 system with event brokers.
 
Operations Planning – We considered IT governance needed for DM Operations. Specifically, we looked into market-grade tools used to support IT service management and how those tools can setup and maintain ITSM processes. We identified a mature open source tool called iTOP ( http://www.combodo.com/?lang=en ) and began prototyping processes in it.
 
Equipment Purchasing Plans – We worked on preparing both the FY15 purchasing plan and the FY16 Acquisition Strategy Document in September. For the FY15 purchase plan, we reviewed the current development infrastructure that is overdue for replacement, investigated items needed for the WAN emulator to test network latency, wrote a plan with general costs and equipment specifications, and delivered the document to AURA for them to begin the procurements. For the FY16 strategy document, we gathered use cases from developers for facilitating upcoming verification tests, considered other development and integration infrastructure needs, created an initial design for the new cluster, and estimated costs of the components. We consulted with NCSA storage, systems, and networking groups for feedback on the design.
 
Sizing model storage costing exercise – In September we finished the exercise of assessing the cost of keeping more than two data releases on spinning disk. This storage would increase the size of the coadd and calibration image caches and the L2 database capacity need, and using LDM-141 (noting that the sizing model costing refresh in August did not fix all inconsistencies attributed to out-of-date technologies) we estimated the cost increase is minimal.
 
The JCC met in September. The discussion centered around specifying operational practices between NCSA and CC-IN2P3, e.g., who is responsible, consulted, and/or informed at each stage of integrating software changes in production. Notes are available on the LSST Confluence ( https://confluence.lsstcorp.org/x/yJV9Ag , https://confluence.lsstcorp.org/x/IpZ9Ag ).
Other management activities in September: we conducted one interview for the Research Programmer position, and our new Research Programmer, James Parson, came onboard during the last week of the month. We finalized our W16 cycle resource-loaded plan and put it in JIRA and PMCS.
 
02C.07.01 Processing Control
 
Orchestration work for verification dataset processing – The orchestration framework “orca” will be used in running verification dataset tests. Orca was used in running data challenges, but hasn’t been used extensively for some time. Thus, in September we focused on upgrading and cleaning up orca to bring it up to date with the way event logging is now implemented in the stack. Documentation was migrated from TRAC to the LSST Confluence and updated ( https://confluence.lsstcorp.org/display/DM/Orchestration ). Finally, we prepared a presentation about orchestration and processing control for the DM Bootcamp.
 
Prototype DRP sequence with DECam data – Much work was accomplished in September in extending, fixing and running the obs_decam package in the LSST stack. Previously obs_decam expected calibrated (instcal) images as inputs, so functionality was added to handle raw, uncalibrated DECam images, ingesting them into a registry and retrieving them with the Butler. Then, a function was added to pipe_tasks to ingest DECam calibration data into a calibration registry and have the Butler retrieve it. Finally, we began running small tests to work through the process of reducing raw DECam data with the calibration files to identify bugs and further changes that are needed to implement raw data processing.  
 
Process Execution Framework – After an early prototype design and a focused meeting in Tucson at the end of September with Gregory and others, the architectural design of the new SuperTask was nearly completed by the end of the month. Notes and a graphical representation of the example demonstration are available at https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=~mcarras2&title=Task+Redesign+Notes .
 
02C.07.02 Infrastructure Services
 
We coordinated with the NCSA CSD team to develop a SOW that involves Identity and Access Management services. The team was introduced to the SUI and Qserv teams and their design ideas and needs. Work will begin in October.
 
02C.07.03 Environment and Tools
 
No activities are planned for the W16 cycle.
 
02C.07.04 Site Infrastructure
 
September work was mostly effort to support operation of the development infrastructure at NCSA. On the existing lsst-dev cluster the sys admin team installed the latest version of devtoolset, upgraded HTCondor to the latest stable version on all servers to make them consistent in preparation for verification dataset running using orca, released a new firewall Puppet module, installed CentOS7.1 on the storage servers, and fixed a drive issue on the database servers. Eight new user accounts were created on the lsst-dev cluster.
 
In preparation for installing new replacement hardware and decommissioning old hardware, they also did a backup of the “temporary” Mac OSX test build and investigated setting up VSphere on Mac Pros.
 
A production OpenStack was deployed in September. Further work included setting up rsbackups, setting up Cinder block storage, and beginning to investigate backup and data replication services like Swift object stores.
 
Planned activities:
 
02C.07.00 Processing Control and Site Infrastructure Management
 
In October we plan to:


·   finalize the FY16 Annual Acquisition Strategy Document and submit it for review. We will simultaneously discuss with NCSA storage, system, and networking teams to plan the procurements and installation timeline and staffing needs.
·   work extensively on specifying the design of the production systems. Under consideration next is a deeper dive into the L1 image ingest and processing system interface between La Serena and NCSA, data access rights at the DAC, and L3 workspace functional requirements.
·   participate in the DM bootcamp.
·   collaborate with CC-IN2P3 via our monthly meeting of the JCC, focusing on NCSA’s FY16 hardware acquisition plans and current infrastructure.
·   continue to develop use cases for operations and participate in planning in the Technical Operations working group, in particular prototyping IT governance processes in iTOP.
·   review candidates for the Research Scientist and Research Programmer positions and conduct several interviews. In October these positions will close.

 
02C.07.01 Processing Control
 
Orchestration work for verification dataset processing – We anticipate a few more modifications to orca to fix the logging mechanism. Emergent work as users start running orca will be supported.
 
Prototype DRP sequence with DECam data – Work to extend the obs_decam package to handle raw data will continue in October. We will investigate missing features in the Instrumental Signature Removal task that currently prevent it from processing raw DECam data, and begin to modify the obs_decam and ISR tasks to handle raw data.
 
Process Execution Framework – Following the meeting in Tucson, we will start to implement the structure of SuperTask in the Process Execution Framework and draft documentation of the design.
 
AP Simulator – In October we will work on documenting the AP simulator.
 
02C.07.02 Infrastructure Services
 
AA & IdM services – We will coordinate biweekly meetings with SUI, Qserv, and others involved in authentication and authorization. By the end of October the CSD team expects to have an initial draft of the LSST IAM Design Document circulated for comments, as well as plans for a technical demonstration.
 
File system policies – Related to and flowing out of the FY16 storage capability design, storage policies like backing up data, identifying managed data (owners, dataset sizes, quotas), and data retention will be proposed.
 
File management technologies – As our new research programmer ramps up in the project, the plan is to have him start prototyping data management mechanisms, such as iRODS, and propose a solution.
 
02C.07.03 Environment and Tools
 
No activities are planned for the W16 cycle.
 
02C.07.04 Site Infrastructure
 
In October our major work will be installing and provisioning new hardware specified in the FY15 purchasing plan. We will also continue to provide administrative support of the development cluster, including monthly maintenance in October. We will work with LSST developers to commission the NCSA OpenStack nebula for devops, including building snapshots of instances running HTCondor.

 


NOAO


Current accomplishments:
02C.08.00 International Communications and Base Site Management Engineering and Integration
 
The month was spent burning vacation hours that have accumulated over the years
 
Planned Activities.
 
02C.08.00 International Communications and Base Site Management, Engineering, and Integration
 
Continue synthesizing Jira epics and stories into PMCS which will be done through Gantt project charts liaising with Kevin Long. The Chilean networking has been updated in Gantt but there is still the International section from Santiago to U.S. to be completed.
 
02C.08.01 Base Center
 
Continue the design of the base Data Center specifications along with the architects.
 
02C.08.03 Long-Haul Networks
 
October will be primarily dedicated to the execution of the RFP for the DWDM end node equipment. We have nine proposals and interviews to read and conduct.
 

Back to top