[ale] System losing memory

[Jim]

JK wrote:
> timothy at meanor.net wrote:
>   
>> Try this: ps -eo pid,rss,vsize,args | sort -n -k 3,3
>>
>> This outputs PID, resident set size (all non-swapped memory process is
>> using), vsize (virtual memory size of process), and command-line & args of
>> each process, sorted by vsize.
>>
>> Do you have any oom_kill messages in any logs in /var/log/ ?  I can't
>> remember if it actually says "oom_kill" in the message, or if it just says
>> "killing process".
>>
>> It's interesting that your system has swap completely utilized, yet almost
>> all memory in use is pagecache.  I recall seeing this before on older Red
>> Hat boxes, but I can't remember what causes this.
>>     
>
> I saw a similar problem on Fedora machines in the (IIRC) Core 3 era or
> so. The cause was a driver for a multi-port serial card (RocketPort, I
> think) which was leaking kernel pages.  (Pages alloc'd by drivers
> don't turn up in process memory maps, since they're not [usually]
> associated with any proc's VM.)
>
> -- JK
>
>   
The system is essentially a ftp/file server with a small c++ app running 
to detect incoming ftp files and add info to a mysql db about the file.  
There is a periodic program to clean up old files and mysql entries.  
There is a Java app running that monitors a remote device and passes 
info to another system via a ssl link. 

I've killed the java app, mysql, and all of the c++ programs but nothing 
frees up memory.  We have pretty much exactly the same programs running 
on multiple systems but this one is the only one experiencing the 
problem.  I'm tempted to scrub the disk and start over.

Thanks,
Jim.