[ale] strange issue with disc space

[Chuck Payne]

On Thu, Jul 30, 2009 at 9:08 AM, Jeff Lightner<jlightner at water.com> wrote:
> One common cause of this is deleting a file while it is still "open".
> That deletes the file name but not the inode.  However, a reboot would
> typically cure that (as would killing the process that had the file
> "open".
>
> Another common cause is putting files in a mount point directory BEFORE
> mounting the filesystem on it.  Such files are part of the parent
> filesystem.  When you later mount a filesystem on top of that directory
> the files in the filesystem mount those in the parent filesystem in that
> directory.
>
> You could check for this by starting system in single user with only /
> mounted.   Check /tmp, /opt, /home, /archive, /var, /usr, /u00, /boot
> and /crs_db to be sure there are NO files there (since none of these
> filesystems are mounted).  If there are any there move them to an
> alternate directory that is not a mount point.
>
> Also mount /usr after doing the above and verify there is nothing in
> /usr/local.   If there are move those off to an alternate directory that
> is not a mount point.
>
> Once you've moved the stuff to alternate directories you can mount all
> the filesystems then move back anything you think you need into the
> mounted filesystems.
>
> You mentioned RAC and I see /crs_db - is that an OCFS or OCFS2
> filesystem?  Those have unique properties as they use DIRECT_IO so you
> may need to research the issue at Oracle's OCFS site if that filesystem
> is your issue and the above didn't resolve it for you.
>
> -----Original Message-----
> From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
> Chuck Payne
> Sent: Thursday, July 30, 2009 8:32 AM
> To: Atlanta Linux Enthusiasts - Yes! We run Linux!
> Subject: [ale] strange issue with disc space
>
> Guys,
>
> I have a question. Have you ever seen disc space just disappear and
> you were unable to find what is using? One of our DB came to me the
> other day pointed out that one of his rac boxes has filled up
>
>                      2.0G  1.7G  214M  89% /
>                      2.0G   88M  1.8G   5% /tmp
>                     1023M  298M  673M  31% /opt
>                      2.0G  203M  1.7G  11% /home
>                      2.0G  297M  1.6G  16% /archive
>                      2.0G  456M  1.4G  25% /var
>                      4.9G  2.7G  2.0G  59% /usr
>                      2.0G   68M  1.8G   4% /usr/local
>                       40G   12G   26G  32% /u00
>                     122M   32M   84M  28% /boot
>                     7.9G     0  7.9G   0% /dev/shm
>                     4.0G  118M  3.9G   3% /crs_db
>
> so I started looking into,  but I can't find any file or dir on the
> root partition that eat space. I have  looked at dmesg and don't see
> anything. Oh, we have rebooted the box and the space didn't clear up.
>
>
> Just an fyi, here are things I did look, please feel free to let me
> know what other things I need to check.
>
> alias ducks='du -chs * | sort -rn | head -11'
> find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{
> print $NF ": " $5 }'  | sort -nrk 2,2
> lsof -a +L1
>
>
> --
> ----------------------------------------
> Old utilities do not die they just slowly fade away!
> -----------------------------------------
> OpenSUSE -- http://en.opensuse.org/User:Terrorpup
> OpenSuSE Ambassador
> OpenSuSE Member
> Skype -- terrorpup
> twitter -- terrorpup
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
>
> Please consider our environment before printing this e-mail or attachments.
> ----------------------------------
> CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.
> ----------------------------------
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
>

I am starting to thinks this is a RAC issue, because we have four RAC
systems ( Each build of 2 Nodes, that means 2 servers), I just got
reported that another issue Node 1 is having the same issue.

I will look into the other items, thanks.

-- 
----------------------------------------
Old utilities do not die they just slowly fade away!
-----------------------------------------
OpenSUSE -- http://en.opensuse.org/User:Terrorpup
OpenSuSE Ambassador
OpenSuSE Member
Skype -- terrorpup
twitter -- terrorpup