1. Data Management

Data Management
For Data Challenge 2 (collection-828), we completed the integration of the Image Subtraction Pipeline with the Middleware. Due to greater than anticipated work in the Pipeline harness, in parallelizing certain portions of the MOPS, and in packaging an automated build for the large number of off-the-shelf middleware packages, as well as planned activity for CoDR, ADASS, IVOA, and DM Face to Face Meetings in September and October, we are approximately 3 weeks delayed for the scheduled execution of DC2 execution in November, 2007. Rather than cut scope and lose some of the research value of DC2 in order to hold schedule, we anticipate completing DC2 by end of December, which is still well in time for all critical R&D milestones.
We conducted the DM Face to Face meeting at Google Mountain View on October 23 – 24. This was a DM All-Hands meeting with additional presentations by Google about relevant technologies. Approximately 50 people attended, with representatives from all DM partners and LSST project management, the Royal Edinburgh Observatory, and Google. We established the basic scope for DC3 and identified areas for exploring open source versions of Google technologies to DC3.
We sponsored and attended the Extremely Large Database (XLDB) conference at SLAC on October 25. This meeting was also very well-attended. This was a unique mix of database academics, providers, industry users, and scientific users. Virtually all attendees rated the meeting as “useful” to “highly useful”. Of particular relevance to LSST DM, we left with a strong sense of how the industry users (eBay, Google, Yahoo, AT&T, others) are already dealing with scaling and operational reliability issues of the kind DM will need to address, and how the suppliers are responding. There will be another meeting in 6 months, at which time we will discuss specific DM use cases and queries and appropriate database primitive functions and query interfaces for astronomy uses.

Back to top