1. Data Management Extended Monthly Report
  2. January 2015
    1. NCSA / University of Illinois
    2. Current accomplishments:
    3. University of Washington
    4. Princeton University and University of California, Davis
    5. IPAC / California Institute of Technology
    6. · Attend the DM AHM at SLAC
    7. · LDM-240 road map revamp
    8. · Finish S15 epics planning
    9. · Make the decision on JavaScript package adoption
    10. · Continue the discussion of APIs with SLAC group
    11. SLAC / Stanford University
      1. 02C.06.02.04 Image Service
      2. · Finish image cutout service V1
    12. LSST Program Office

Data Management Extended Monthly Report

Back to top

January 2015

NCSA / University of Illinois

Current accomplishments:

02C.07.00 Processing Control and Site Infrastructure Management

In January the information security program was completed. An Acceptable Use Policy that describes allowed usage of equipment owned by AURA for the LSST project was developed, incorporating policies from existing AUPs at NCSA, XSEDE, and Blue Waters. An Incident Response Plan and Information Classification Policy were drafted. Effort was also placed in developing plans and risk tables for each of the master security plan sub-projects, including Data Management, Project Management, Education and Public Outreach, Camera, and Telescope and Site. Following a week-long trip to Tucson to discuss these sub-plans and meet with members of the Telescope and Site team, the information security program documents were packaged and put under standard version control. Don was given the position of temporary Information Security Officer, and in February we will find a permanent person to hire for this position.

There was considerable discussion in January towards finalizing the procurement contract that will cover hardware purchases for equipment that will be housed at NCSA and at the base site in Chile. The concern is that the conditions of the contract must be consistent with existing purchasing system at the University of Illinois and in compliance with Federal regulations. It appears we are close to a consensus on the ownership of equipment, number of quotes needed to purchase new equipment, and the frequency at which the sizing model will be updated.

In January we spent time restructuring our W15 cycle epics. We also began planning for the Summer15 cycle. The outstanding procurement contract noted above, as well as a still undetermined start date for our two new hires, are major unknowns that could significantly affect our plans for the next six months.

02C.07.01 Processing Control

Most of January was spent on refactoring event services. The event base class was refined to remove obsolete terms and make it more flexible with properties of ActiveMQ messages. Unit tests were updated to be consistent with the current unit test framework and duplicate tests were removed. Modifications were made to message IDs to make them more meaningful (capturing host name, process ID, and local ID) and therefore more filterable in the Event Receiver. Finally, a series of regression tests were run in the Alert Production simulator and Orca to ensure changes made to ctrl_events did not cause unforeseen problems. All expected work was completed and was placed in review at the end of the month. Depending on the outcome of the review, minimal if any work will be done on this in February.

In January, another version of the state diagram of the nightly compute simulator, showing job progress through a run, was constructed.

02C.07.02 Infrastructure Services

In January, effort was placed on file system and file management, in particular setting up an inventory of spinning disk storage. Towards the end of the month the NCSA team met to discuss the form this inventory should take, considering what storage is included, the level of inventory granularity, and the user interface for this data (e.g., a simple web page or a more complex UI). It is important that this effort work together with the new NFS servers that the infrastructure team installed in January (see below), as well as the current data classification activities. It was decided that the best path forward would be to stand up the new NFS servers with a new management structure and to begin cataloging and managing new data, and then later set up a process of migrating old data to the new servers under the new structure. Questions about data integrity, data management hierarchy, and data replication were also considered.

Work emerged in the month of January to prepare for data from the IR camera. Accounts were setup for the Telescope and Site team, a new area on the storage servers was created for data, http access was enabled for the T&S team, and a new backup process was created to store the data in the Blue Waters archive.

02C.07.03 Environment and Tools

Evaluation and testing on the NCSA Innovative Systems Lab (ISL) OpenStack testbed continued in January. A read-only volume was created using the OpenStack cloud management node to store LSST test data. Experiments attempting to extend the volume size while the volume was still attached to an instance demonstrated that modifications to the volume can only be made if the volume is first detached from the instance. The cloud management node command line client was also used to make a snapshot of a data volume and subsequently 'clone' additional volumes of input test data from that snapshot. These duplicate data volumes may be attached to individual OpenStack instances, providing a mechanism for working with the 'a data volume may only be attached to one instance' limitation that exists in current OpenStack releases.

Investigation continued in January on the ability to run HTCondor Central Manager using an OpenStack instance. Initial tests verified that a worker node on an LSST resource/VM can be configured to "glide-in" to the central manageron the NCSA OpenStack cloud by utilizing the public IP number of the central manager as the effective hostname. An issue was observed in the case of submitting Condor-G jobs to a Globus gatekeeper. It appeared that job submissions would not complete due to communication issues between remote machines and the central manager in OpenStack. DNS configurations changes were made following advice from the HTCondor team, which appear to have remedied the situation. Thus, the central manager running on the OpenStack instance is fully functional, demonstrating that beyond having the ability to provision HTCondor worker nodes in the OpenStack cloud, we can also spin up job submission/manager nodes.

02C.07.04 Site Infrastructure  

In January, due to the pending finalized purchasing agreement, the infrastructure team at NCSA accomplished no work towards ordering new hardware, and none is expected in the coming month. With the final NFS storage servers delivered at the very end of December, the team worked on their installation and configuration. These new storage servers will have three systems in production and one as a test system or spare to allow for doing maintenance or reconfiguration of the three production systems. A basic OS configuration of all four servers, including networking and privileged accounts, was performed.

After some assessment of Intelligent Platform Management Interface (IPMI), it was determined that all necessary remote functions and monitoring capabilities are possible using IPMI access, and therefore IPMI was deemed a reliable alternative to IP-KVM currently used for remote access. We will not need to upgrade the KVMs as previously thought.

A few LOE tasks emerged during January and were addressed. One account on the ISL OpenStack testbed was set up. The remainder of the LSST Linux servers were configured to log time zone information to work around the request for system-wide logging in UTC. Virtual disk was added to one of the VMs being used to run Docker. To assist the needs of the Database team in loading data automatically into Qserv, the sys admins at NCSA set up a new rsync module on the lsst-rsync server.

Planned activities:

02C.07.00 Processing Control and Site Infrastructure Management

In February it is expected that we will have a draft of amendments to the NCSA sub award for procurements.

We will continue to consider our S15 cycle plans in February. Two DM members from NCSA will travel to SLAC for the DM All-Hands meeting mid-month; this meeting will hopefully guide our planning within the larger picture of LSST-DM goals.

Additionally, effort towards developing a draft of the base site data center requirements is expected.

02C.07.01 Processing Control

Another iteration of feedback will confirm whether this version of the state diagram of the nightly compute simulator is sufficient or whether another draft will be constructed in the coming month.

Due to schedule slipping in December and the dependency of other work on completing the refactoring of event services, tasks originally planned to begin this month are now expected for next month. These will include developing a handler to emit events for log messages using the Apache log4cxx logging framework, and, if time, implementing a mechanism in Event Services for tasks executing in parallel to communicate.

02C.07.02 Infrastructure Services

We identified areas that will need further discussion in the coming month, such as defining our “unit of data management” and policies for usage thresholds and enforcing quotas.

02C.07.03 Environment and Tools

In February, focus of OpenStack evaluation will be on automation. Work thus far has been accomplished utilizing the Horizon GUI interface and the command line tools, but in the coming month the use of Python scripts to automatically launch an OpenStack instance and install LSST software will be explored.  

02C.07.04 Site Infrastructure  

In February, configuration will continue and performance tests using various operating systems. If all goes well, deployment of the servers will begin.

Work in February will focus on setting up a network and bastion host for IPMI and documenting how it can be used with the LSST infrastructure.

It is not likely that our new hires will arrive before the end of February, so we will reassess the delivery of sizing model critique.

University of Washington

Current accomplishments:

02C.03.00 Alert Production Management Engineering and Integration

Interviewed 3 candidates for the DM Scientist job at UW. Prepared for the DESC meeting that was in the first week of Feb.

02C.03.05 Application Framework for Exposures

The camera team has signed off on the tool for reading geometric information from the test stand data. The code has been reviewed by Jim C.

02C.03.08 Astrometric Calibration Pipeline

Russell finished implementing the Tabur(2007) algorithm in the LSST fork of the stack. Unit tests are passing and a default implementation using the astrometry.net index files as reference format is in place. The new interfaces allow other solvers and catalog formats to be plugged in.

Planned activities:

02C.03.00 Alert Production Management Engineering and Integration

February will also require planning for and attending the DM all hands meeting at SLAC. We will also be planning for the Summer 2015 cycle and iterating with Kevin on the plan.

02C.03.08 Astrometric Calibration Pipeline

Simon and Russell will remove astrometry.net dependency from the meas_astrom package. This will require implementing a basic reference catalog that does not depend on the astrometry.net index file format.

Princeton University and University of California, Davis

This report covers work carried out in FM4 of FY15 in the Data Release Production group (staff at Princeton plus Price and Gee working remotely).

Current accomplishments:

02C.04.00 Data Release Production Management Engineering and Integration

Through this month Bosch & Swinbank invested significant effort in planning activities for the group. There are three major activities:

 • Re-planning Winter 2015, to take account of cost & schedule variances that have opened up due to training and ramp-up time for new hires;

 • Planning activities to be performed during Summer 2015;

 • Consider long-term roadmap activities in the context of LDM-240 (in collaboration with Krughoff at the University of Washington).

02C.04.01 - Application Framework for Catalogs

Major effort focused on the remaining tasks for the measurement framework overhaul. These include final adjustments to the framework itself (DM-464, -1125, -1218, -1659 and -1771) and converting algorithms from the old framework to the new one (DM-280, -1333, -1456 and -1783). In addition, updates were performed to the afw::table system to support the new functionality (DM-964).

In addition, priority was given to work on the “calibration and ingest” framework (DM-1074) which blocks progess by the database group (DM-210). A prototype of this code was available by the end of the month

02C.04.03 - PSF Estimation

No work was carried out under this WBS during this month.

02C.04.06 - Object Characterization Pipeline

The iterative process of adapting our pipeline to run with Hyper Suprime Cam data continues. Significant progress has been made and basic functionality is available, but a protracted code review process and many minor issues mean that this work is still ongoing (DM-245).

Planned activities:

All team members will attend the Data Management All Hands Meeting at SLAC in mid-February.

Planning for Summer 2015 will be completed and locked in to our schedule.

Focus on completing the major milestones required to close out the Winter 2015 cycle. Primarily, this means the completion of the measurement framework overhaul, and getting the pipeline to a stage where it can perform basic reductions on data from Hyper Suprime Cam, which is essential for future testing & commissioning.

IPAC / California Institute of Technology

Current accomplishments:

The IPAC group accomplishments included:

·   David Ciardi gave the group a good summary of the LSST data products.
·   David Ciardi and Xiuqin Wu attend AAS in Seattle, met with Simon to discuss the alert system. There are many areas need clear definitions
·   Level 3 data products discussion led us to believe we should start the workspace definition and
·   design early since it involves at least IPAC, SLAC, NCSA, SQuaRE.
·  SUI requirements study and discussion led us to believe we need to bring the visualization functions
·   discussion to DM AHM.
·  Xiuqin Wu attended the UML training for 4 days.
·  Continued weekly meeting with SLAC group, mainly to discuss data access APIs.
·   Prepared for DM AHM in SLAC next month.

Planned activities:

·   Attend the DM AHM at SLAC

·   LDM-240 road map revamp

·   Finish S15 epics planning

·   Make the decision on JavaScript package adoption

·   Continue the discussion of APIs with SLAC group

SLAC / Stanford University

Current accomplishments:

02C.06.00 Science Data Archive and Application Services Management Engineering and Integration

·  Prepared for DM AHM @ SLAC

·  Planned database team work for the Summer 2015 cycle

·  Organized weekly Qserv and Data Access meetings

·  Prepared for XLDB 2015 Conference (cont)

·  SLAC related: Delt with issues related to SLAC reorganization (accounting, support, office space)

02C.06.01.01 Catalogs Alerts and Metadata

·   Continued building the Metadata Store

02C.06.01.02 Image and File Archive

· Implemented FITS header crawler
· Coordinated implementation of the V1 of the form for collecting user input data
02C.06.02.01 Data Access Client Framework
· Continued work on Butler V2
02C.06.02.03 Query Services
· Continued work on Qserv refactoring (DM-1030)
· Continued w ork on Management of distributed Qserv databases and tables (DM-1 036)
· Prepared test data set and started running Qserv as a service for SUI tests
· Almost-finished the implementation of C++ geometry and porting to Qserv (DM-214)
· Released Qserv 201501 and accompanied documentation
· Tested Qserv on Scientific Linux 7
02C.06.02.04 Image Service
· Defined and implemented RESTful interfaces (v0) for Data Access Services
Planned activities:
02C.06.00 Science Data Archive and Application Services Management Engineering and Integration
· Orchestrate DM AHM (local logistics)
· Finish planning database team work for the Summer 2015 cycle
· Setup SLAC accounts for MREFC work
· Continue preparing for XLDB 2015 Conference
02C.06.01.01 Catalogs Alerts and Metadata
· Finish Data Ingest
· Finish building Metadata Store prototype
· Implement Data Ingest (DM-210)
· Prepare and deliver Web Data Access Services demo at
02C.06.01.02 Image and File Archive
· Wrap up V1 of the form for collecting user input data
02C.06.02.01 Data Access Client Framework
· Finish building Butler V2
02C.06.02.03 Query Services
· Finish C++ geometry (DM-214)
· Finish work on Qserv refactoring (DM-1030)
· Finish integration testing (DM-1031)
· Improve Qserv stability (DM-1061)
· Release Qserv 201502 with documentation

02C.06.02.04 Image Service

· Finish image cutout service V1

LSST Program Office

DM Project Management and Control

Current accomplishments:

The DM Project Manager:

·   Continued to support NSF Cooperative Agreement risk/contingency negotiations. Attended meeting with Project Office and Carol Wilkinson of the NSF to clarify the approach to estimating contingency.
·   Under the leadership of Don Petravick at NCSA, and with the support of the NSF Center for Trustworthy Scientific Cyberinfrastructure, Lee LeClair of Ephibian, German Schumacher, Kian-Tat Lim, and Gregory Dubois-Felsmann, the IT Security Program (aka Cyber-security plan) was delivered to the NSF on time.
·   Continued recruiting and hiring activities: opened the DM Deputy System Architect (internal posting) and DM SQuaRE Software/Documentation Engineer positions.
·   Continued work with AURA on MREFC contracts for all DM lead institutions. The SLAC, Princeton, NCSA, and REUNA contracts are signed. UW, IPAC, and FIU sub award contracts have completed final round of comments with AURA contracts and institution counterparts and are awaiting signature.
·   Continued development of draft Memorandum of Agreement with IN2P3 for operations.  Comments from NCSA are expected and a final round of edits will be made, before submission to the LSST Corp. Board subcommittee. An executive coordination meeting in Lyon between LSSTPO, NCSA and IN2P3 has been scheduled for March 5. A meeting in Lyon between NCSA and IN2P3 for implementation coordination has been scheduled April 22.
·   Continued development of MOA for Brazilian contributions to operations (Networks). The MOA has been briefed to the responsible agencies, the Ministry of Science and Technology (MCTI) and the Sao Paolo Research Foundation (FAPESP). A letter of support enabling the MOA and allowing for signature by the Brazilian RENs RNP and ANSP is in process.
·   Initiated Winter 2015 plan update and Summer 2015 detailed plan.
·   Planned the Data Management All Hands Meeting at SLAC February 9 - 13.
·   Served on the panel of the US Atlas Operations Review at BNL.
·   Briefed the AMCL on DM status.
·   Coordinated approach to updating the Calibration Plan with Robert Lupton, Mario, Juric, Gregory Dubois-Felsmann, Chuck Claver, and George Angeli. Dr. Lupton and I identified a list of candidates to perform near-term analysis tasks in support of the plan, and Dr. Lupton is in the process of contacting them.

Planned activities:

The DM Project Manager will:

·   Continue to support NSF Cooperative Agreement Risk/Contingency negotiations. Develop correlations in DM plan in PMCS.
·   Continue recruiting and hiring: conduct interviews for DM SQuaRE Software/Documentation Engineer and Deputy System Architect.
·   Complete work with AURA on MREFC contracts for remaining DM lead institutions.
·   Continue development of draft Memoranda of Agreement with IN2P3 and Brazil.
·   Complete Summer 15 planning
·   Conduct the Data Management All Hands Meeting at SLAC February 9 - 13
·   Present Data Management status at AMCL meeting
·   Continue updates to IT Security Program, assign IT Security Officer

DM Science

Current accomplishments:
Hiring remained the focus of January activities, in addition to travel to support project-wide activities such as the Project Science Team and AAS meetings.
The DM Project Scientist interviewed three additional candidates for the Level 1 DM science lead (position located at UW). The top candidate has been identified and the offer has been made. We're expecting to hear back from the candidate by the end of February.
The DM Project Scientist also interviewed five candidates for the DM Analysis scientist (recently renamed to LSST Data Systems Fellow) postdoc position, and
identified two outstanding candidates. Work is under way to identify non-project funds that could be used to hire both, thus helping LSST development (as well as diversity).
The DM Project Scientist also took part representing LSST at the AAS meeting in Seattle, and the PST meeting in Tucson.
Planned activities:
In February, the DM Project Scientist's primary focus will remain on completion of staffing up, activities related to the DM All Hands meeting, as well as representing DM in project-level or project-related meetings (DESC and PST). The DM Project Scientist also expects to spend a significant fraction of time evaluating PhoSim for its continued suitability as a tool for DM work.
DM System Engineering

Current accomplishments:

Activities completed by the DM System Architect include:

·   Assisted with computer security document
·   Worked with SUI/Data Access on image cutout and REST API design
·   Defined lines between file systems, file management, metadata, databases
·   Attended HEP Software Foundation workshop, discussed lessons learned
·   Attended Kahnfest
·   Prepared for DM All Hands Meeting
·   Described LSST alerts for Symmetry magazine article
·   Worked on refinement of New Butler
·   Discussed potential reuse of LSST software with Nao Suzuki of IPMU

Planned activities:

The DM System Architect will:

·   Help conduct DM All Hands Meeting
·   Start refining Base Center design
·   Discuss status of Sizing Model and note action items
·   Refresh team on development practices and tools
·   Assist with deep dive into Data Release algorithms and dataflows
·   Deliver talk on LSST data movement at CrossConnects Cosmology workshop
·   Attend OCS Middleware Workshop to define details of OCS communication
·   Work to define details of Camera DAQ protocols
·   Participate in Chilean Networks Acquisition review
·   Continue work on New Butler

DM Science Quality and Reliability Engineering (SQuaRE)

Current Accomplishments:

02C.01.02 Management

·   Josh Hoblitt onboarding: JH officially started DM work in mid-January, as SQuaRE’s first new addition

·   Robyn Allsman off-boarding: RA retired from LSST-DM, we wish her all the best with her new pursuits

02C.01.02.03 L3 Toolkit (Productized Stack)

·   Automatic VM provisioning for stack release integration tests: JH developed a Vagrant-based infrastructure for doing test builds required for release preparation

02C.07.03.01 Software Development Tools (Developer Support)

·   JIRA modifications for Simulations group: FE supported the Sims groups as they merged their two largest JIRA projects and developed a new workflow

·   Github migration preparation work: FE & JH made some modifications to ease the transition of our repositories to Github

·   Test removal of git package from new installation process: FE verified that we do not need to distribute our own version of git, as it is not necessary for newinstall.sh to complete successfully

05C Education and Public Outreach

FE & JH spent 1 week on an EPO Sprint

 Planned Activities:

·   Migration to git

·   Recruitment for Software/Documentation Engineer position

·   DM All Hands meeting

·   S15 Planning

·   JH to attend SCALE

 DM Applications, Middleware, and Infrastructure

Current accomplishments:

Work continued towards end of February completion in all areas of Winter 2015 applications and middleware. Planning for Summer 2015 started with all teams identifying next round of features and tests against the LDM-240 Key Progress Metrics.

Planned activities:

·  Continue Winter 2015 sprints per JIRA and PMCS plans
·  Complete Summer 2015 planning
·  Continue needs analysis for first acquisitions of Development and Integration clusters.
·  Continue Base Data Center design as updates to LSE-77 ICD and a more detailed lower level document.
·  Conduct LSST Chilean Networks Acquisition Review on February 24

Back to top