[ale] strange issue with disc space

[Jeff Lightner]

One common cause of this is deleting a file while it is still "open".
That deletes the file name but not the inode.  However, a reboot would
typically cure that (as would killing the process that had the file
"open".

Another common cause is putting files in a mount point directory BEFORE
mounting the filesystem on it.  Such files are part of the parent
filesystem.  When you later mount a filesystem on top of that directory
the files in the filesystem mount those in the parent filesystem in that
directory.

You could check for this by starting system in single user with only /
mounted.   Check /tmp, /opt, /home, /archive, /var, /usr, /u00, /boot
and /crs_db to be sure there are NO files there (since none of these
filesystems are mounted).  If there are any there move them to an
alternate directory that is not a mount point.   

Also mount /usr after doing the above and verify there is nothing in
/usr/local.   If there are move those off to an alternate directory that
is not a mount point.

Once you've moved the stuff to alternate directories you can mount all
the filesystems then move back anything you think you need into the
mounted filesystems.

You mentioned RAC and I see /crs_db - is that an OCFS or OCFS2
filesystem?  Those have unique properties as they use DIRECT_IO so you
may need to research the issue at Oracle's OCFS site if that filesystem
is your issue and the above didn't resolve it for you.

-----Original Message-----
From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
Chuck Payne
Sent: Thursday, July 30, 2009 8:32 AM
To: Atlanta Linux Enthusiasts - Yes! We run Linux!
Subject: [ale] strange issue with disc space

Guys,

I have a question. Have you ever seen disc space just disappear and
you were unable to find what is using? One of our DB came to me the
other day pointed out that one of his rac boxes has filled up

                      2.0G  1.7G  214M  89% /
                      2.0G   88M  1.8G   5% /tmp
                     1023M  298M  673M  31% /opt
                      2.0G  203M  1.7G  11% /home
                      2.0G  297M  1.6G  16% /archive
                      2.0G  456M  1.4G  25% /var
                      4.9G  2.7G  2.0G  59% /usr
                      2.0G   68M  1.8G   4% /usr/local
                       40G   12G   26G  32% /u00
                     122M   32M   84M  28% /boot
                     7.9G     0  7.9G   0% /dev/shm
	             4.0G  118M  3.9G   3% /crs_db

so I started looking into,  but I can't find any file or dir on the
root partition that eat space. I have  looked at dmesg and don't see
anything. Oh, we have rebooted the box and the space didn't clear up.


Just an fyi, here are things I did look, please feel free to let me
know what other things I need to check.

alias ducks='du -chs * | sort -rn | head -11'
find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{
print $NF ": " $5 }'  | sort -nrk 2,2
lsof -a +L1


-- 
----------------------------------------
Old utilities do not die they just slowly fade away!
-----------------------------------------
OpenSUSE -- http://en.opensuse.org/User:Terrorpup
OpenSuSE Ambassador
OpenSuSE Member
Skype -- terrorpup
twitter -- terrorpup
_______________________________________________
Ale mailing list
Ale at ale.org
http://mail.ale.org/mailman/listinfo/ale
 
Please consider our environment before printing this e-mail or attachments.
----------------------------------
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.
----------------------------------