Apparently v12.5.3 esd 7 64bit on Linux has a problem with the backup server. It throws a signal 11 and dies. Nice feature, huh?
The problem is that esd 7 introduced a 64bit backupserver binary and still uses the old compat-libcstd++-5 library. The 64bit version of the library is somewhat flakey. Using the 32bit binary from esd 6 is the workaround (make sure to include the esd7 version of sybmultbuf too).
The workaround is to set LD_POINTER_GUARD to 0 in the RUN_<SERVER>_BS file just prior to the backupserver command line. Now, Sybase fixed the ASE binary (dataserver) atleast for the 12.5.x codeline but missed the backupserver and sybmultbuf (64bit versions).
According to Sybase Techsupport, the problems with GLIBC 2.4 will be fixed within ASE 12.5.4 ESD #6!