[ale] A petabyte here, a petabyte there ...

[James P. Kinney III]

On Sun, 2007-07-29 at 11:13 -0400, Chris Farris wrote:
> 2TB/.001usec ?

That is the nuclear interaction timing from particle impact to the
excitation that generates the actual signal that is the data collected.
> 
> I'd love to hear how they manage to capture that.

To the best of my recollection there is a flotilla of several thousand
machines that that each capture a portion of the raw data torrent. My
friends PhD program works on those machines to do the first layer filter
to and chuck out most of the non-interesting events. I think there is
some special hardware that may be able to write directly from the
scintillation cell electronics to RAM on each of the first stage
machines.  I do know that the wiring for the stage 1 systems was a
particular challenge as they were immediately adjacent to the hot zone.
The radiation shielding may protect the box but not the wire leading to
it. I saw the setup before it was up and running. Shielding was concrete
blocks 5'x5'x10'. All wiring access holes had to be offset at 45 degrees
from the face of the block to prevent direct radiation pathways. 
> 
> I just crossed the TB threshold on my home network. Yay Sol11 & ZFS!
> 
> Chris
> 
> James P. Kinney III wrote:
> > A friend of mine works at Fermi lab. The D0 project generates ~2TB every
> > of data in about .001 usec. This volume gets filtered down to about 10
> > GB through a huge pile of systems (my friend wrote some of the physics
> > event filtering code) and finally get dumped to an array of tape drives.
> >
> > This process repeats every 20 seconds they have beam time.
> >
> > On Fri, 2007-07-27 at 17:17 -0400, Greg Freemyer wrote:
> >   
> >> I've had 2 different client conversations in the last 2 days.
> >>
> >> All I can say is "A petabyte here, a petabyte there, and pretty soon
> >> you're talking real data".
> >>
> >> Seriously, one claimed to have 3 PB of unstructured data (office
> >> files, logs, etc.).  The other one was smaller.  They only had 1 PB of
> >> unstructured data.
> >>
> >> I admit to being a little taken aback, especially the first time.  I
> >> guess it is time to add exabyte to my repertoire.  That way I can say,
> >>
> >> "Oh.., only .1% of a Exabyte, sure our solution can handle that with
> >> no problem. (cough, cough) I was afraid you would need something
> >> large."
> >>
> >> Are others starting to see numbers like that at the large Enterprise
> >> level?  (Both of these companies had multiple NAS units at multiple
> >> sites.  I think the biggest single NAS I heard of is 200TB so far.  I
> >> really had gotten away from these huge systems for while.  I guess I
> >> need a refresher about what's happening out there in the "enterprise
> >> world".)
> >>
> >> Greg
> >> -- 
> >> Greg Freemyer
> >> The Norcross Group
> >> Forensics for the 21st Century
> >> _______________________________________________
> >> Ale mailing list
> >> Ale at ale.org
> >> http://www.ale.org/mailman/listinfo/ale
> >>
> >>     
> >> ------------------------------------------------------------------------
> >>
> >> _______________________________________________
> >> Ale mailing list
> >> Ale at ale.org
> >> http://www.ale.org/mailman/listinfo/ale
> 
> -- 
> Chris Farris				chris at vitalpowers dot com
> The Exercise of Vital Powers		404 806 1403
> http://www.vitalpowers.com
> 
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
> 
-- 
James P. Kinney III          
CEO & Director of Engineering 
Local Net Solutions,LLC        
770-493-8244                    
http://www.localnetsolutions.com

GPG ID: 829C6CA7 James P. Kinney III (M.S. Physics)
<jkinney at localnetsolutions.com>
Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190 ADC3 829C 6CA7
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part