Status reports for LSST Corp

Report for August 15, 2006 (covers period from 7/16/07 to 8/15/07)

Status reports for LSST Corp.

Data Management Team

Report for August 15, 2006 (covers period from 7/16/07 to 8/15/07)

Summary of Progress
Work activity, whether it reached a milestone or not

U.Pitt/CMU & UA-CS
· These teams have finished their work and are no longer required to report.

UW (submitted on August 14, 2007 by Nicole Silvestri)
Andy B.
· Added MINUIT to build system; created fitting class that merges MINUIT and Function.
· Back on track after build problems introduced by the mwi/fw split. Solved by checking out the trunk for mwi and fw and using those instead of the release versions.
· Coded up three tests cases, each of which involves comparing an input image to a convolved version and trying to reconstruct the convolution kernel: 1) compare input image to itself; should get a delta function 2) compare input image to version convolved with a spatially invariant Gaussian kernel 3) compare input image to version convolved with a spatially varying Gaussian kernel.
· Wrote code to create an Alard/Lupton convolution kernel.
· The image subtraction code is finished aside from QA (python unittests will be written by NS) and the ongoing discussion with RHL about using matrix inversion or some other technique.

Nicole
· sent Linearization breakdown and DC2 plan updates to Cristina for DC2 telecon (Aug 7) but we decided as a DC2 group to hold off on working on this until Image Subtraction code is nearly finished.
· Wrote and tested test code in C++ for Exposure Class and WCS class (makeExposure.cc and wcsTest.cc in tickets/36/examples). Tim has fixed issues with WCS Class (reassigned Ticket 112 to Tim and sent comments on Ticket #93) and has merged these fixes with the trunk so NS can now test WCS functionality in Exposure class. Exposure Class passes all basic tests except those dealing with the 'getSubExposure' method.
· Submitted Exposure Class (Ticket 36) for preliminary review to RO. Still needs some work but RO can start review while further enhancements are being implemented.
· Received and started working on preliminary review comments from RO.
· Successfully synchronized Exposure Class code with UML model (thanks to some good advice from Tim A.). Added a page to the Twiki on how to do this properly: http://lsstdev.ncsa.uiuc.edu:8100/trac/wiki/EA_Sync .
· submitted bug fix to Ticket #32 - EA color map issue.
· typed up and posted weekly minutes from our local LSST group meetings (http://www.astro.washington.edu/becker/LSST/local/Minutes/ ).
· prepared monthly progress report for Cristina

Russell:
· Finished Ticket 25 (Kernel class), got it reviewed by RHL, updated based on the review and merged to trunk.
· Started on Ticket 123 (python interface and test cases for Kernel class). The Function interface and test cases are done. The Kernel stuff is about 1/4 done.
· Partway through the python interface and test suite for the Kernel classes and convolve function. Finished AnalyticKernel and FixedKernel. Stuck on the convolve function and have asked Robert for help. Cannot compile the code that swigs the convolve function due to complaints about MaskedImage::operator= Also stuck on LinearCombinationKernel (and have asked Robert for help): cannot figure out how to create a LinearCombinationKernel in Python; can't seem to format the input list of kernels in a way that it recognizes. Fixing this will probably require some enhancements to the python interface.
· Began the Python interface for the Exposure class (NS will write the test cases), but ran into the same problem as the convolve function mentioned above: complaints about MaskedImage::operator=.
· Began review of the Exposure Class (Ticket 36). Sent preliminary review comments to NS.
· Requested info on Policy file from AB & NS: wrote and sent Policy File needs/recommendations of UW group to LSST-Data.
· Resolved and closed Ticket #32 - EA color map issue.
· Helped NS and AB write their software.

Lynne & Simon:
· LJ: on vacation Aug 06-13.
· put TALCS raw data and calibration frames for raw data on dev.lsstcorp.org
· wrote up alert report
· wrote up photometric variation report
· 80% complete evaluating and comparing cronosNEO and cronos92 for NEO completeness, using synthetic NEO population obtained from Alan Harris, for NASA report with Simon Krughoff
· organized Gemini GMOS-S Y band sky brightness observation program with Simon Krughoff, Marcel Bergman, Perry Gee, Kem Cook and Chris Stubbs
· preparing dithering evaluation tool with SK - 50% complete, waiting on input from Tony Tyson and Phil Pinto
· preparing evaluation of Y, u and g filters for science council face-to-face meeting

NCSA (submitted on August 14 by Chris Cribbs and David Gehrig, and on August 16 by Ray Plante and Greg Daues) (Steve Pietrowicz is on vacation until August 19,2007) (ray Plante on vacation August 6-10)
· The NCSA team hosted a 2-day meeting with Sweeney, Kantor, and Cutri.
· Ray continued moderating the DC2 telecons and the MW WG.
· Ray and Cristina are using the DC2 plan to track progress toward getting ready for Data Challenge 2. The plan was refined to reflect work at a finer granularity.
· Ray finished the split of support classes (mwi) from the fw package.
· Ray worked on Policy class. In particular, he wrote up a design description in Trac (PolicyDesign). http://lsstdev.ncsa.uiuc.edu:8100/trac/wiki/PolicyDesign
· Ray worked on review of DataProperty.
· Ray wrote document describing the design of the pipeline framework. http://lsstdev.ncsa.uiuc.edu:8100/trac/wiki/PipelineFramework
· Chris installed and configured new LSST cluster and installed SAN disk on LSST cluster (he was on vacation for 1 week)
· Greg worked on the documentation and coding of the lsst.dps package for the Distributed processing services. These efforts were fed by requirements and documentation specified by astronomy team members working on Association, MOPS, and Image Processing and Detection pipelines. Specifically focused on the means by which input parameters are delivered and made available to the application Stages of these Pipelines by utilizing incoming Events, the Pipeline/ Slice middleware objects, and the Queue & Clipboard objects that belong to pipeline Stages. A Trac page of documentation and a first draft of code are now written to deliver information within events to application Stages.
· David added new packages to the build system and debugged builds.
- an upgrade to wcslib
- added mpich2 package
- an upgrade to sconsUtils
- mysql client package
· David installed MySQL 5.0.45 on the lsst cluster (lsst10).
· David did some minor maintenance on the Subversion system, creating a new area for contributed non-pipeline software and giving new users access. He also did some minor maintenance on the Trac system, giving new users access to the system.
· David moved the test data files to lsst9 in preparation for making the templates and object catalog. David determined, with Andy Becker, the header information necessary for the test data images to be processed properly.
· Ongoing tasks for David: maintained/debugged build system, including updated builds (w/ assistance from Ray) and maintained TRAC site.

LSST/NOAO (submitted on August 13, 2007 by Robyn Allsman)
(returned from vacation on July 31, 2007)

DM Standards
· Over Summer leave: Ray Plante took over Logging and implemented a new protocol taking into consideration the NCSA 'event' requirements.
· Documented the approved method of documenting an LSST code using Doxygen.
· Reimplementing Exception handler since the new (over summer leave) implementation of DataProperty provides a new user API. Prototype of new implementaiton is 95% done. It is usable now but needs 3 user helper routines for completion.
· Continuing to update the LSST Error Handling Document (http://lsstdev.ncsa.uiuc.edu:8100/trac/wiki/ErrorHandling ) on issues, standards, and testing related to Exception Handling.

PRECURSOR DATA Prep - not relevant to DC2
· Assisting Jon Smillie ANUSF/Australia install the precursor data into a DB and provide a VO-enabled front-end.
My role: point him to my work done a year ago which includes:
1. definition of DB schema
2. scripts for formatting the data for CSV ingest
3. scripts for ingesting into MySQL and PostGreSQL DB
Over summer leave: JonS installed a VO interface to the ingested data.

· Following is on indefinite hold:
o Talked with Ray about what needs to be done to integrate the LSST-reformatted MACHO data into the LSST data archive. Project scientists will determine which subset of the data should be ingested into the LSST Precursor DB. This detail doesn't need to wait on any preliminary task.
· Status: Same as last month:
o images containing (at least) one truncated amp image were reprocessed at ANUSF to remove the partial amp image and then WCS reprocessed. The xfer of reprocessed images to NCSA was prematurely stopped when size of new images was found to be too short for any amp image to be included.

LLNL (submitted on August 21, 2007 by Don Dossa)
· Review a few solid state disk products: I have been in contact with Vion, FusionIO, and Exegy.
· Updated costs and tech trends section of NSF proposal for the CoDR.
Goal: Find money to attend the CoDR.

Database Group (submitted on Aug 14, 2007 by Jacek Becla for J Becla/SLAC, K-T Lim/SLAC, Ani Thakar/JHU, Maria A. Nieto-Santisteban/JHU, Kem Cook/LLNL, Serge Monkewitz/IPAC, Andy Hanushevsky/SLAC)
Serge
· finished AP entirely/partly in the applications tests
- basis for actual AP code, code reviewed by K-T
· design work on how to integrate AP with the pipeline framework
- specifically, how to break the AP into stages, how to map FOVs to pipelines
· decided on a method for running multiple FOVs in parallel (one pipeline per FOV)
· design work on a simplified AP (most data to be retained in the database)
· began implementing AP proper
- match code (both distance based/ellipses) done (needs more testing)
- support for maintaining AP state in shared memory code nearly complete
- goal is to implement both the simplified AP and the all-in-application design (can reuse a lot of code to do this)
- about to begin implementing Stage classes.

K.T.
· worked on persistence design and two iterations of implementation (with a third to come)
· worked on the design of the Association Pipeline and helped clarify how overall pipeline framework functions
· implemented database ingest/dump utilities for chunks of Object data for the AP
· setting up an instance of the LSST development environment at SLAC
· Attended meeting in UCDavis
· worked on packaging CORAL for DC2
· coordinated DataAccWG telecons

Maria
· Discuss with Jacek work assignment to review/compare official LSST Schema and DC2 Schema. Write summary of the discussion and task assignment
· Compare LSST precursor archive specification with LSST official schema.
· Map FITS keyword headers against the LSST official schema.
· Document what was found, what wasn't found, and issues related to type compatibility.
· Call attention about over normalization, classes should not always map directly to database tables.
· Study the precursor archive specification and create a schema that should allow for both storing and retrieving FITS header metadata for DC2.
· Explore the Exposure related classes to see what I could do in the schema. http://lsstdev.ncsa.uiuc.edu:8100/trac/browser/DC2/fw/tickets/36/include/lsst/fw . I didn't find much.
· Modify Source table according to Andy Becker's changes/additions. http://lsstdev.ncsa.uiuc.edu:8100/trac/browser/DC2/imageproc/tickets/7/include/lsst/fw
· Respond to K-T about the mySQL precision issue
· Respond to Serge about the ObjectPhotoZ issue
· Comment on simplification of AS pipeline
· Participate in telecons

Ani
· Discussed precursor schema with Maria
· Phonecons
· Reviewed and discussed simplified AP proposal with Maria

Andy Hanushevsky
· Started to look at mysys in mysql to see how to "statically" integrate xrootd in a way that the mysql folks can accept as a permanent patch.

Jacek
· on vacations July 15 - Aug 19
· checking emails and addressing urgent issues

GMU (submitted on Aug 13, 2007 by Kirk Borne)
GMU staff supports the LSST project in the areas of data management, data products, community science database access, and EPO (education/public outreach). Specific monthly activities during the reporting period included:
· GMU participated in online and telecon discussions regarding NSF proposals, including plans for submitting a CI-TEAM proposal by August 27 to NSF. GMU provided information about the CI-TEAM program, as learned at the NSF CI-TEAM proposers' workshop in July.
· GMU solicited information from NASA database experts related to their use of large-scale databases, sent this information to SLAC member of the LSST DM team, and then asked these NASA database folks to consider participating in the XLDB (Extremely Large Databases) workshop at Stanford University in October.
· GMU reviewed the GalaxyZoo and Galaxiki web portals for the LSST EPO team, and compared their implementation and goals with that of the planned AstroDAS portal for LSST community data access.
· GMU gave a public lecture in Baltimore in July, which focused on "Science@HOME" concepts, including SETI@Home and LSST@Home. The audience was asked for feedback about their potential participation in some of our proposed LSST EPO community data access portals, including LSST@Home, CN^2, Adopt-a-Patch-of-Sky, AstroDAS, and more. The feedback was very strongly positive. Everyone was excited to hear about these plans and looked forward to participating.
· GMU completed updates to the Map-A-Planet DataSheet that was initiated by the GMU-led Project AstroData team at last year's DLESE Data Services Workshop in Tucson. The DataSheet can be found at: http://serc.carleton.edu/dev/usingdata/datasheets/PDS_MAP_Moon.html . This is one module in the Carleton College "Using Data in Education" science education resource portal.

Jeff Bartels (submitted on Aug 14, 2007)
· Third iteration of DataProperty redesign completed, implemented, tested.
- Resulting change impacts to all post-split mwi and fw code implemented, tested.
- Changes to UML model underway.
· Assumed role of mwi package-meister.
- Deployment of mwi 1.1 release in process.

Detailed Progress
Details of above

Major Accomplishments
Only significant breakthroughs, issues resolved

· UW team:
o Submitted Exposure Class (Ticket 36) for review. Received preliminary review comments.
o Kernel Class completed (Ticket 25) and closed.
o All Image Subtraction Use Cases (Ticket 07) have been coded up.
o Completed draft reports on alerts and photometric measurement errors with changes in atmosphere / filter.

Milestones Achieved
Only major tasks in project plan

Objectives for the Next Period
What you expect to accomplish by next month

UW
Andy B.:
· Finish last diffim imageproc use case computeSpatiallyVaryingPSFMatchingKernel; start with test code.
· Finish and verify the three tests cases (see AB section above).
· Compose email about the conditions required for matrix inversion to reliably do the job for determining a kernel that will psf match a postage stamp of a science image to a template image

Nicole:
· Remaining tasks associated with Ticket 36 include:
- test WCS class implementation in wcsMatch function.
- finish test of 'getSubExposure' function as soon as Tim adds WCS functionality for this to WCS Class (close Ticket #111) before next DC2 telecon (Aug 21).
- address Exposure (in)completeness issues (see AppWG minutes from Aug 13) and review comments.
- Merge Ticket36 (after final review) to trunk.
· Start writing Python unittests for Exposure Class (need Python interface [see RO section above] to test) and for AB's Image Subtraction code (will eventually need Python interface to test for this as well).
· Update sequence diagrams and synchronize changes from final code version with EA model for Exposure Class.
· Discuss Persistence with Jeff Bartels before next DC2 telecon (Aug 21).

Russell:
· Will work on generating images for testing convolution and writing test cases, thus preparing for having a working python interface (or for writing the test suite for convolve in C++)
· Finish ticket 123 and merge.
· Continue to provide invaluable help with code to AB and NS.

Lynne & Simon:
· finish NEO evaluation (LJ & SK)
· package dithering evaluation tool for science groups (LJ & SK)
· filter evaluation details for face to face meeting
· document ubercalibration procedure and write code

NCSA
· Ray and Cristina will continue to use the DC2 plan to monitor progress and to make adjustments to the schedule and/or scope as necessary.
· Ray will continue to moderate the DC2 and MW WG telecons.
· Ray will complete Policy class and check in Logging changes, update EA model accordingly.
· Ray will complete FileFormatter class for use in DC2
· Ray and other NCSA team members will be involved in preparations for CoDR presentations.
· David will continue preparing the test data: making templates; making catalog data; ingesting in precursor archive
· David will begin coordinating DC2 integration: determining status of all components and moving components to lsst cluster
· Greg: will continue on the documentation and coding of the lsst.dps package for the Distributed processing services. Within the next time installment, testing of the code with events communicating between Pipelines will be done. This will certify that that the required coordination of Pipelines in DC2 can be achieved. Other issues that will be addresses include :
1. incorporation of Policy into the lsst.dps,
2. execution of pipelines on the DC2 cluster platform,
3. integration of MPICH into the software stack
4. merging of the current lsst.dps development within a "ticket" into the svn trunk.
· Chris will be reviewing Document 646 so it can be updated where needed.
· Chris will install Lustre on LSST cluster
· Chris is working on a plan for testing data movers with Arun.

LLNL
· Don will be getting an NDA presentation from Vion in Sep/Oct.
· Understand latest requirements from Chile on OCS/TCS since it looks like it needs to be hooked into the DM pipeline somehow.
Goal: Arrange a trip to CTIO bringing German, Ron, Chris, Mike Huffer together to understand if DM architecture need to change.

LSST Corp
DM Standards
· Will serve as new logging protocol implementation reviewer when Ray's implementation is complete.
· Need to find out if the approved method of documenting an LSST code using Doxygen has been by reviewed by rest of committee.
· Exception handler scheduled for completion in 1 week.
· LSST Error Handling Document Scheduled for completion in 1 week.
PRECURSOR DATA Prep - not relevant to DC2
· I will perform general sanity check on new VO interface.
· Ask DM Software manager what datasubset should be ingested.
· JonS examining error logs to determine if there is a problem. Could just be that all amp images were determined to be truncated in original image.

DB group
· coordinate implementation of Association Pipeline and Database Services for DC2 [all]
· code review by K-T, and check preliminary AP code into svn. I will coordinate with K-T to see about getting stuff done while I'm on vacation (Aug 22 - Sep 6). [Serge]
· implementation of Database Services [KT]
· work on interfacing application code with database schema [all]
· continue work on xldb workshop [Jacek]
· continue work on mysqld + xrootd [AndyH]
· integrate persistence architecture with rest of code including check-in to subversion, code reviews, and creation of Trac components [KT]

GMU
· GMU will attend the ASP education/outreach conference in Chicago, will participate in LSST-related outreach sessions, and will participate in the "Telescope Networks for Education" workshop at Adler Planetarium.
· GMU will work on the NSF CI-TEAM cyberinfrastructure-education proposal, to be submitted by LSSTC, GMU, and others.
· GMU will develop a report that summarizes the review of the LSST sample queries from the assembled responses of the LSST education / outreach partners.
· GMU will report on the science-specific sections of the LSST Database Schema to the LSST Galaxies Collaboration Team.

Jeff Bartels
· Conclude development of Persistence in conjunction with K-T Lim.

Problems Encountered and Solutions Being Pursued
Budget or schedule variances, technical issues, management issues

· UW Team
- AB: The mwi/fw merge set AB back more than a week, as he was unable to operate efficiently with the build system such a mess. This was resolved on Aug 7, and he am now a week or so behind on code development.
- RO: Spent a lot of time fighting problems with the build system. Build system issues have been resolved with exception of a few queries to RHL on remaining minor issues.
- NS: Stuck for about a week waiting for proper merge to the build system of WCS Class and wcslib changes. Had some minor issues with 'eups distrib' on local system. Build system issues have been resolved. Code development has resumed.
- LJ: computer queries of millions of objects (OpSim history) against millions of objects (NEO observations) take a long time to complete. Need to look into status of UW cluster.
· Jeff Bartels: I continue to spend significant amounts of time overcoming build problems associated with the use of the build system. These have been a combination of improper usage, unexpected results from documented/proper usage, synchronization of changes by multiple parties, etc. Am going to invest the time to do deeper study of the available documentation on eups and the build system, in hopes that I can avoid further delays caused by these problems
· David Gehrig: The build system continues to be a bottleneck. I had hoped to be generating templates by now but haven't had the time because problems with the build system. Because of the centrality of the build system, anyone's showstopper becomes my showstopper.
· Don Dossa: UC Davis funding process is complete and I am getting paid from LSST money via Davis. The only problem, and this appears to be unsolvable, is that my money from Davis runs out this month because of the dates on my appointment at Davis.
Goal: The new goal is to get my MLA agreement between LLNL and Davis renewed for the next FY. This whole funding situation is getting very tedious.