[ale] Dealing with really big log files....

[Michael B. Trausch]

On Mon, 23 Mar 2009 09:54:35 -0600
JK <jknapka at kneuro.net> wrote:

> This is the only good idea so far. It's the only one that
> doesn't involve reading the entire file up to the data of
> interest, which is just... not a very smart thing to do,
> when you have some idea where the data is, in a file
> approximately the same size as the known universe.

True enough, but...

You'd have to know how long each line is to use that well, or
accurately.  If the lines are variable-length, as I suspect that might
be, then you've really no way to determine via 'dd' what line you're
starting on, just the "block" number, which would only really be good
if you know about what blocks you need from the file.

So, line-centric solutions fit, unless the lines are all fixed-length
(or, padded to a fixed length to make block calculations possible).
Otherwise, the only way to accurately figure out what to feed as
parameters to dd is to use a line-centric method to figure it out in
the first place... which is why I advocated compressing and splitting
and working on the chunks.  The only flaw in mine was that I should
have used -c or -l in my example, instead of -b.  Habit got me on that
one.

	--- Mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://mail.ale.org/pipermail/ale/attachments/20090323/e7512709/attachment.bin