If you are going to be using ASE 12.5 or ASE 15 with the upcoming Red Hat Enterprise Linux 5.0, there are two CRs that you need to know about:

  1. CR 450004 – XP Server and Backupserver (64bit)
  2. CR 432482 – ASE 15

The problem is the result of using the jmp() and longjmp() functions in the GNU C Library version 2.4 (glibc 2.4).

setjmp() and longjmp() are useful for dealing with errors and interrupts encountered in a low-level subroutine of a program. setjmp() saves the stack context/environment in env for later use by longjmp(). The stack context will be invalidated if the function which called setjmp() returns.

Starting in glibc 2.4, the jmpbuf is unavailable due to a security exploit known for several years.  Restricting access to the jmpbuf was an effort by the glibc maintainers to ‘close this hole’ by forcing the application writers to use the sigsetjmp() and siglongjmp() instead.

The problem with using sigsetjmp() and siglongjmp() is that they aren’t thread safe.  This is supposed to be fixed in glibc 2.6 according to a contact at Sybase but Sybase *does* have a workaround for this issue and will be putting it into the ASE v15 codeline when they perform the RHEL 5 certification sometime next year.

I wasn’t able to find much information the supposed exploit.  I only found a few newsgroup posts that didn’t provide any details.

Michael Peppler maintains the ASE on Linux FAQ.  It contains a workaround for various distributions of Linux:

Installing/Running ASE on systems with glibc 2.4.x (Fedora Core 5, etc)

ASE (12.5.x, 15.x) will not run on a system with glibc 2.4.x (that’s Fedora Core 5 and later, among others). Here is a work-around, supplied by “Chris” in sybase.public.ase.linux:

Symptoms of the problem are ASE’s failing to build when you run the installer, or “Segmentation Fault” when trying to start an ASE instance. The Seg Fault happens right after Engine ) is brought online in the error log.
If you are running Fedora Core 5 or 6
You will need to set the environment variable LD_POINTER_GUARD to 0, in your sybase .profile, as follows:

export LD_POINTER_GUARD=0

Alternatively, just export this from the shell. If you haven’t built any ASE’s yet, you’ll need to export this before running the installer, otherwise ASE will fail to build. If you are trying to boot an already installed ASE, then put it in your .profile / .bash_profile or export it as an environment variable before starting ASE.
If you are running Ubuntu / Kubuntu “Edgy”
You will need to set the environment variable LD_POINTER_GUARD to 1 in your sybase .profile, as follows:

export LD_POINTER_GUARD=1

I googled around a bit trying to find out what the LD_POINTER_GUARD thingie did and ran across a posting / blog where it talked about insome glibc versions it’s backwards … RedHat just made a change I guess and that’s why you have to set it to “0″ for Fedora and “1″ for Ubuntu / Kubuntu / Debian (glibc 2.4). Same info applies if you are building a new ASE.
If you are running Suse 10.1 You will need to set the environment variable LD_POINTER_GUARD to 1 in your sybase .profile, as follows:

export LD_POINTER_GUARD=1

Same discussion as for Ubuntu.
I’ve tried all of the above they all work with the workaround listed. Happy ASE-ing! I’ve tested all of the above distros (sorry it took me so long, was on the road a lot over the last several weeks) and they allow ASE to boot; the workaround should be valid for ASE 15.x and 12.x (prolly even for 11.x if you wanted to try).

UPDATE: Workaround for all current versions of ASE at http://froebe.net/blog/2008/06/26/how-to-install-and-run-sybase-adaptive-server-enterprise-1254-and-150x-on-ubuntu-linux-804-jeos-hardy-heron-using-chroot-environment/

« »