A few weeks ago I mentioned having troubles with direct i/o and ASE, ASE 15.0.1 esd 2 on Linux with direct i/o woes. Guess what? I solved the problem!
It ended up being the driver for the Areca ARC-1110 PCI-X RAID Card. Using the driver obtained from Areca (ftp://ftp.areca.com.tw/RaidCards/AP_Drivers/Linux/DRIVER) I was able to reliably reproduce the problem. When I upgraded the kernel to 2.6.20 which includes a driver for the ARC-1110 RAID card, the problem went away. 
There appears to be a memory leak in the driver provided by Areca under certain circumstances such as ASE 12.5x, 15.x using O_DIRECTIO with lots of I/O. This memory leak will cause ASE to be killed by the kernel as the ASE process (dataserver) will appear to have eaten up all available memory. The driver included in the v2.6.19 and higher kernels don’t seem to have this problem in my testing.
I’ve asked Sybase TechSupport to “Please create a solved case explaining that if the ASE process is killed by the kernel (it will be in /var/log/messages), then quite likely the cause is a driver issue.“
« NAS disk init issues ASE 12.5.x, 15.0.x on Windows XP, 2k, 2k3, Vista Linux Gaming (FPS / Strategy) »
How’s Sybase TechSupport replied on that case?
I encountered Memory Leak by ASE Dataserver processes, unable to find out what’s wrong with it? Can you share your experiences on this? Thanks.
Sybase TechSupport created the Solved Case as I requested. As far as general memory leaks go with Sybase ASE, it is extremely rare for the dataserver to leak memory and is more likely the cause of libraries and/or drivers (RAID card driver in my case).