From buland@usgs.gov Mon Oct 22 16:26:44 2001
--============_-1208329314==_ma============
Content-Type: text/plain; charset="us-ascii"; format="flowed"
X-Sun-Content-Length: 10117
Dear Lind and David,
I've thought a lot about the Menlo Park meeting since I headed
home. You asked for some simple recommendations and got an earful of
development philosophy, sales propaganda, and a list of messy
problems (some in vast detail). For me, I think part of the problem
was that I didn't have a clear enough idea of what you were trying to
accomplish until late in the meeting. However, in retrospect, I
think that the kernels of several concrete recommendations can be
worried out of the discussion.
Restating what I think you were trying to get to, I think you were
looking for recommendations on:
1) How best to meet the combined Menlo Park/Berkeley requirements
using existing building blocks and
2) How best to bridge between existing Earthworm Classic front ends
and the NCEDC archive (using the Berkeley schema) with limited
software development staff.
What I heard at the meeting, though not in these exact words, was
that the recommendation depends on your priorities. If you are
primarily interested in the short term, the recommendation would
clearly be to adopt the Trinet software, get a working system in
place quickly, and work around its limitations as best you can. On
the other hand, if you are primarily interested in the long term,
then the recommendation is that you need to shop for a solution in
the same way that you would shop for any other major
hardware/software subsystem.
To elaborate, a short-term solution would consider how much time
and effort would be needed to meet most of your requirements with
less consideration given to meeting all of your requirements. As
Doug (Neuhauser) has pointed out, Trinet already bridges between an
Earthworm "classic" front end and the NCEDC archive and satisfies
most of your requirements. Splitting the DBMS functionality between
the archive and the collators solves the problem of two real-time
systems and one archive rather neatly. The remaining problems are:
1) adapting the Menlo Park shake map feeder, 2) supporting real-time
tasks such as moment tensor and finite fault estimates, and 3)
providing the functions needed to synchronize the separate processing
systems.
The Menlo Park shake map feeder is an issue because the problem of
integrating real-time and non-real-time accelerometer data in a
variety of formats has already been solved in Menlo Park, but using
the Earthworm DBMS. Apparently, the problem of integrating late
accelerometer data into the Trinet system has not yet been dealt
with. The problem of supporting moment tensor and similar algorithms
that require access to raw waveform snippets after an event is
located is easy if you can wait for the request cards to be
processed, but requires direct access to the wave pool otherwise.
Synchronization issues will require gatekeeper functionality,
divergence monitoring and resolution, and system state-of-health
monitoring across the Menlo Park-Berkeley systems (and will probably
have to be done locally no matter what system you choose).
A long-term solution is more difficult because the total cost of
ownership must be considered over the projected life of the system.
The cost of purchasing, installing, and adapting the hardware and
software, including adding missing functionality, will be important.
The time required to install and adapt the system also can be thought
of as a cost unless deadlines are impending. The cost of maintaining
the system, including the robustness of the hardware and software
(how often it needs maintenance), availability of long-term support
from the designers, and the need for specialized services (e.g., a
database administrator) will also be a factor.
The evolution of the system also must be considered. There should
be a clear (and economical) path for expanding the system as the
network grows and it should be possible to adapt to new processing
needs and cooperative partners. The risk of external economic
factors shortening the life of the system (e.g., Oracle going broke)
also should be considered. Finally, although harder to access, it is
also useful to know if the system is still under active development
and if that development is likely to be of future benefit.
In this area, the Trinet solution is somewhat less attractive
based on my interpretation of Rob Clayton and Doug Given's comments.
It appears that Trinet is already using hefty (and presumably
expensive) multiprocessor Sun servers (presumably with
correspondingly expensive Oracle licenses) and they are already
encountering some performance problems (e.g., inserting amplitude
measurements). Adapting Trinet to support needed tasks appears to
require working around fundamental architectural limitations (e.g.,
moment tensors).
Trinet is a mature system and has a lot to offer. However, it was
designed for a very specific environment and is already showing signs
of "work hardening" (e.g., complex applications directly manipulating
the DBMS make the system dangerous to modify). Also, the (currently
unsupported) list of planned development tasks are mostly cleaning up
remaining problems. It doesn't sound as though significant continued
development of Trinet in Southern California is very likely.
Finally, while it is clear that the Southern California folks would
like to see a Trinet system in Northern California (for political,
psychological, and practical reasons), it is hard to predict the
level of support that Berkeley-Menlo Park might expect from the
south. In the recent past, the Caltech people have been reluctant to
interact with other networks because of overwhelming problems at
home. It remains to be seen if this will change.
The Earthworm DBMS provides a striking contrast to Trinet. It is
currently not a very mature system, but it will continue to be under
active development for the foreseeable future. The Earthworm DBMS is
not as central to the real-time processing system and is currently
able to employ much less expensive platforms and licenses. It is
hoped that, this strategy will keep the DBMS well behind the
"bleeding edge" where it might become a bottleneck limiting system
expansion. Earthworm already supports tasks that require the
automatic processing of raw waveforms after initial processing (e.g.,
Ml processing from wave servers) and, of course, it supports the
near-real-time shake map.
We believe that the Earthworm DBMS is built on a very solid
foundation (and, yes, this is where all that philosophical junk comes
back to haunt you). The Earthworm DBMS was meant to be a bulletproof
complement to the Earthworm classic front end. From the beginning,
the philosophy was to adopt industrial design and implementation
practices including the transactional schema orientation, a
reasonable degree of normalization, and requiring access through well
developed application program interfaces (APIs).
What does this really mean? Well, without getting into a lot of
detail (and sounding way too much like a sales pitch), it means that
the Earthworm DBMS was designed from the ground up to be robust, but
flexible. In particular, it was meant to be adaptable to a variety
of seismic network environments, to evolve with network needs, and to
be long lived (longer than any specific implementation such as
Oracle). As I pointed out at the meeting, support is more
problematical. It seems clear that Earthworm will continue to be
supported at least until ANSS can develop something else
(realistically 3-4 years). If the present funding situation
continues, which seems likely, Earthworm may well be all that the
ANSS can afford to develop even in the longer term.
So what does all this mean for Berkeley and Menlo Park? The
Earthworm database is specifically designed to support real-time and
near-real-time (e.g., the collator) functions, but is not designed to
support archive and long term distribution functions. That is, the
Earthworm database is intended to be a component in a larger system
that would generally include two different databases with different
organizations and functions. Therefore, using the Earthworm database
as a building block is tantamount to adopting a hybrid
Earthworm/Trinet database system. This will inevitably increase
initial system cost (because of the need to interface them). What
you must assess is whether the possible benefits of a hybrid system
(e.g., robustness, flexibility, longevity, etc.) are sufficient to
lower the overall system cost over the projected system life.
As you can see, this isn't much of a recommendation (in keeping
with the outcome of the meeting). Hopefully, it will help to make
the choices somewhat more clear cut. The only thing I can add is
that I fully intend to continue the Earthworm DBMS development
because I have concerns about the cost, flexibility, scalability, and
longevity of the Trinet system. Developing an interface between
processing center databases and archive center databases seems to be
an integral part of the ANSS plan. My plan is to accept that model
and get on with it.
PS. I started this letter the Friday after the meeting, but just got
a chance to finish is now (having survived my trip to Boston and back
being exposed to nothing more lethal than non-dairy creamer). I hope
it is still relevant to the on-going discussion.
--
Ray Buland
USGS/NEIC
Golden, CO, USA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
USNSN Project Manager U.S. Geological Survey
buland@usgs.gov Box 25046, Stop 967
office: 303-273-8414 Denver Federal Center
FAX: 303-273-8450 Denver, CO 80225, USA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--============_-1208329314==_ma============
Content-Type: text/html; charset="us-ascii"
X-Sun-Content-Length: 12777
Dear Lind and
David,
I've thought a lot about the Menlo Park meeting since I
headed home. You asked for some simple recommendations and got
an earful of development philosophy, sales propaganda, and a list of
messy problems (some in vast detail). For me, I think part of
the problem was that I didn't have a clear enough idea of what you
were trying to accomplish until late in the meeting. However, in
retrospect, I think that the kernels of several concrete
recommendations can be worried out of the discussion.
Restating what I think you were trying to get to, I think you were
looking for recommendations on:
1) How best to meet the combined Menlo Park/Berkeley requirements
using existing building blocks and
2) How best to bridge between existing Earthworm Classic front ends
and the NCEDC archive (using the Berkeley schema) with limited
software development staff.
What I heard at the meeting, though not in these exact
words, was that the recommendation depends on your priorities.
If you are primarily interested in the short term, the recommendation
would clearly be to adopt the Trinet software, get a working system in
place quickly, and work around its limitations as best you can.
On the other hand, if you are primarily interested in the long term,
then the recommendation is that you need to shop for a solution in the
same way that you would shop for any other major hardware/software
subsystem.
To elaborate, a short-term solution would consider how
much time and effort would be needed to meet most of your requirements
with less consideration given to meeting all of your requirements.
As Doug (Neuhauser) has pointed out, Trinet already bridges between an
Earthworm "classic" front end and the NCEDC archive and
satisfies most of your requirements. Splitting the DBMS
functionality between the archive and the collators solves the problem
of two real-time systems and one archive rather neatly. The
remaining problems are: 1) adapting the Menlo Park shake map feeder,
2) supporting real-time tasks such as moment tensor and finite fault
estimates, and 3) providing the functions needed to synchronize the
separate processing systems.
The Menlo Park shake map feeder is an issue because the
problem of integrating real-time and non-real-time accelerometer data
in a variety of formats has already been solved in Menlo Park, but
using the Earthworm DBMS. Apparently, the problem of integrating
late accelerometer data into the Trinet system has not yet been dealt
with. The problem of supporting moment tensor and similar
algorithms that require access to raw waveform snippets after an event
is located is easy if you can wait for the request cards to be
processed, but requires direct access to the wave pool otherwise.
Synchronization issues will require gatekeeper functionality,
divergence monitoring and resolution, and system state-of-health
monitoring across the Menlo Park-Berkeley systems (and will probably
have to be done locally no matter what system you choose).
A long-term solution is more difficult because the total
cost of ownership must be considered over the projected life of the
system. The cost of purchasing, installing, and adapting the
hardware and software, including adding missing functionality, will be
important. The time required to install and adapt the system
also can be thought of as a cost unless deadlines are impending.
The cost of maintaining the system, including the robustness of the
hardware and software (how often it needs maintenance), availability
of long-term support from the designers, and the need for specialized
services (e.g., a database administrator) will also be a factor.
The evolution of the system also must be considered.
There should be a clear (and economical) path for expanding the system
as the network grows and it should be possible to adapt to new
processing needs and cooperative partners. The risk of external
economic factors shortening the life of the system (e.g., Oracle going
broke) also should be considered. Finally, although harder to
access, it is also useful to know if the system is still under active
development and if that development is likely to be of future
benefit.
In this area, the Trinet solution is somewhat less
attractive based on my interpretation of Rob Clayton and Doug Given's
comments. It appears that Trinet is already using hefty (and
presumably expensive) multiprocessor Sun servers (presumably with
correspondingly expensive Oracle licenses) and they are already
encountering some performance problems (e.g., inserting amplitude
measurements). Adapting Trinet to support needed tasks appears
to require working around fundamental architectural limitations (e.g.,
moment tensors).
Trinet is a mature system and has a lot to offer.
However, it was designed for a very specific environment and is
already showing signs of "work hardening" (e.g., complex
applications directly manipulating the DBMS make the system dangerous
to modify). Also, the (currently unsupported) list of planned
development tasks are mostly cleaning up remaining problems. It
doesn't sound as though significant continued development of Trinet in
Southern California is very likely. Finally, while it is clear
that the Southern California folks would like to see a Trinet system
in Northern California (for political, psychological, and practical
reasons), it is hard to predict the level of support that
Berkeley-Menlo Park might expect from the south. In the recent
past, the Caltech people have been reluctant to interact with other
networks because of overwhelming problems at home. It remains to
be seen if this will change.
The Earthworm DBMS provides a striking contrast to
Trinet. It is currently not a very mature system, but it will
continue to be under active development for the foreseeable future.
The Earthworm DBMS is not as central to the real-time processing
system and is currently able to employ much less expensive platforms
and licenses. It is hoped that, this strategy will keep the DBMS
well behind the "bleeding edge" where it might become a
bottleneck limiting system expansion. Earthworm already supports
tasks that require the automatic processing of raw waveforms after
initial processing (e.g., Ml processing from wave servers) and, of
course, it supports the near-real-time shake map.
We believe that the Earthworm DBMS is built on a very
solid foundation (and, yes, this is where all that philosophical junk
comes back to haunt you). The Earthworm DBMS was meant to be a
bulletproof complement to the Earthworm classic front end. From
the beginning, the philosophy was to adopt industrial design and
implementation practices including the transactional schema
orientation, a reasonable degree of normalization, and requiring
access through well developed application program interfaces
(APIs).
What does this really mean? Well, without getting
into a lot of detail (and sounding way too much like a sales pitch),
it means that the Earthworm DBMS was designed from the ground up to be
robust, but flexible. In particular, it was meant to be
adaptable to a variety of seismic network environments, to evolve with
network needs, and to be long lived (longer than any specific
implementation such as Oracle). As I pointed out at the meeting,
support is more problematical. It seems clear that Earthworm
will continue to be supported at least until ANSS can develop
something else (realistically 3-4 years). If the present funding
situation continues, which seems likely, Earthworm may well be all
that the ANSS can afford to develop even in the longer term.
So what does all this mean for Berkeley and Menlo Park?
The Earthworm database is specifically designed to support real-time
and near-real-time (e.g., the collator) functions, but is not designed
to support archive and long term distribution functions. That
is, the Earthworm database is intended to be a component in a larger
system that would generally include two different databases with
different organizations and functions. Therefore, using the
Earthworm database as a building block is tantamount to adopting a
hybrid Earthworm/Trinet database system. This will inevitably
increase initial system cost (because of the need to interface them).
What you must assess is whether the possible benefits of a hybrid
system (e.g., robustness, flexibility, longevity, etc.) are sufficient
to lower the overall system cost over the projected system
life.
As you can see, this isn't much of a recommendation (in
keeping with the outcome of the meeting). Hopefully, it will
help to make the choices somewhat more clear cut. The only thing
I can add is that I fully intend to continue the Earthworm DBMS
development because I have concerns about the cost, flexibility,
scalability, and longevity of the Trinet system. Developing an
interface between processing center databases and archive center
databases seems to be an integral part of the ANSS plan. My plan
is to accept that model and get on with it.
PS. I started this letter the Friday after the meeting, but just got a
chance to finish is now (having survived my trip to Boston and back
being exposed to nothing more lethal than non-dairy creamer). I
hope it is still relevant to the on-going discussion.
--
Ray Buland
USGS/NEIC
Golden, CO, USA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
USNSN Project
Manager U.S.
Geological Survey
buland@usgs.gov Box 25046, Stop 967
office:
303-273-8414 Denver
Federal Center
FAX:
303-273-8450 Denver,
CO 80225, USA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--============_-1208329314==_ma============--