[ale] A petabyte here, a petabyte there ...

[List]

Hey, let's not be judgmental....  If one assumes quantum superposition 
as a data transport mechanism...

But, seriously, from my days at one of our esteemed national 
laboratories, I can say that the data capture rates for high energy 
experiments is well beyond what makes your head spin if you think in 
terms of commercial data rates or data quantities.  I have papers from 
nuclear research done over 30 years ago that references terabytes of 
data taken from experiments lasting less than 2 seconds.  Apply Moore's 
law, and you get an idea of what might be possible today. (p.s. don't 
use a TI calculator to figure it out - the exponent does not go high 
enough).

God, I feel old.

Tom

James P. Kinney III wrote:
> On Sun, 2007-07-29 at 11:13 -0400, Chris Farris wrote:
>   
>> 2TB/.001usec ?
>>     
>
> That is the nuclear interaction timing from particle impact to the
> excitation that generates the actual signal that is the data collected.
>   
>> I'd love to hear how they manage to capture that.
>>     
>
> To the best of my recollection there is a flotilla of several thousand
> machines that that each capture a portion of the raw data torrent. My
> friends PhD program works on those machines to do the first layer filter
> to and chuck out most of the non-interesting events. I think there is
> some special hardware that may be able to write directly from the
> scintillation cell electronics to RAM on each of the first stage
> machines.  I do know that the wiring for the stage 1 systems was a
> particular challenge as they were immediately adjacent to the hot zone.
> The radiation shielding may protect the box but not the wire leading to
> it. I saw the setup before it was up and running. Shielding was concrete
> blocks 5'x5'x10'. All wiring access holes had to be offset at 45 degrees
> from the face of the block to prevent direct radiation pathways. 
>   
>> I just crossed the TB threshold on my home network. Yay Sol11 & ZFS!
>>
>> Chris
>>
>> James P. Kinney III wrote:
>>     
>>> A friend of mine works at Fermi lab. The D0 project generates ~2TB every
>>> of data in about .001 usec. This volume gets filtered down to about 10
>>> GB through a huge pile of systems (my friend wrote some of the physics
>>> event filtering code) and finally get dumped to an array of tape drives.
>>>
>>> This process repeats every 20 seconds they have beam time.
>>>
>>> On Fri, 2007-07-27 at 17:17 -0400, Greg Freemyer wrote:
>>>   
>>>       
>>>> I've had 2 different client conversations in the last 2 days.
>>>>
>>>> All I can say is "A petabyte here, a petabyte there, and pretty soon
>>>> you're talking real data".
>>>>
>>>> Seriously, one claimed to have 3 PB of unstructured data (office
>>>> files, logs, etc.).  The other one was smaller.  They only had 1 PB of
>>>> unstructured data.
>>>>
>>>> I admit to being a little taken aback, especially the first time.  I
>>>> guess it is time to add exabyte to my repertoire.  That way I can say,
>>>>
>>>> "Oh.., only .1% of a Exabyte, sure our solution can handle that with
>>>> no problem. (cough, cough) I was afraid you would need something
>>>> large."
>>>>
>>>> Are others starting to see numbers like that at the large Enterprise
>>>> level?  (Both of these companies had multiple NAS units at multiple
>>>> sites.  I think the biggest single NAS I heard of is 200TB so far.  I
>>>> really had gotten away from these huge systems for while.  I guess I
>>>> need a refresher about what's happening out there in the "enterprise
>>>> world".)
>>>>
>>>> Greg
>>>> -- 
>>>> Greg Freemyer
>>>> The Norcross Group
>>>> Forensics for the 21st Century
>>>> _______________________________________________
>>>> Ale mailing list
>>>> Ale at ale.org
>>>> http://www.ale.org/mailman/listinfo/ale
>>>>
>>>>     
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Ale mailing list
>>>> Ale at ale.org
>>>> http://www.ale.org/mailman/listinfo/ale
>>>>         
>> -- 
>> Chris Farris				chris at vitalpowers dot com
>> The Exercise of Vital Powers		404 806 1403
>> http://www.vitalpowers.com
>>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://www.ale.org/mailman/listinfo/ale
>>
>>     
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://www.ale.org/mailman/listinfo/ale