Sybase

We’ve been upgrading a number of Sybase ASE databases from the end-of-life’d 12.5.x to v15.0.3 esd 4 the last few weeks.   My coworker (Degang He) ran into an interesting error when the upgrade process was failing on sysprocedures.

Database ‘my_db’: beginning upgrade step [ID 26]: alter table (table sysprocedures)
Command has been aborted.

Msg 3461, Level 20, State 1:
Server ‘my_server’, Line 1:
Database ‘my_db’: upgrade could not install required upgrade item ’26′. Please refer to previous error messages to determine the problem. Fix the problem, then try again.

Msg 3451, Level 20, State 1:
Server ‘my_server’, Line 1:
Database ‘my_db’: upgrade has failed for this database. Please refer to previous messages for the cause of the failure, correct the problem and try again.

Msg 3454, Level 20, State 1:
Server ‘my_server’, Line 1:

Database ‘my_db’: ASE could not completely upgrade this database; upgrade item 1501 could not be installed.

We looked on Sybase’s Solved Cases and Google but didn’t find anything that would really fit the problem.  So we called up Sybase Technical Support.  Somdev Sharma of Sybase was able to determine that this was a bug in the ASE 15x codeline and provided a workaround:

CR #643188:

Title Online database failed with 3461 error during upgrade , ‘max parallel degree’ >1

Workaround:

Before online the database, disable parallel sort via sp_configure ‘max parallel degree’,1

Sanitized:

Online database failed with 3461, 3451 and 3454 errors during upgrade sysprocedures table (upgrade  item ’26′). If ‘max parallel degree’ > 1, then please disable parallel sort with sp_configure ‘max parallel degree’, 1.

After disabling parallel sort, the database was upgraded successfully.

Can’t call method “value” on an undefined value at /usr/share/perl5/IO/Uncompress/RawInflate.pm

Reinstalling Perl and all the related ubuntu packages doesn’t fix it, nor does removing the ~root/.cpan dir.  There is an easy workaround though.

Make a note of the packages that it will remove, then tell it to go ahead. It will remove these packages!

Start cpan, and type in

. When it is complete, reinstall the ubuntu packages that you removed earlier (for example):

That’s it. Your cpan is working again

repro:
1) create a sql udf in a db
2) run sp_tables, you will see the UDF as a “SYSTEM TABLE”

DBADEV1.sybase_dba.1> sp_tables;
table_qualifier table_owner table_name table_type remarks
--------------- ----------- ---------- ---------- -------
sybase_dba dbo DATE_FORMAT SYSTEM TABLE NULL

In the above example, the DATE_FORMAT UDF is the one at Porting MySQL’s date_format function to Sybase ASE 15.0.2

I’ve opened a case with Sybase to get this fixed – I’ll post the CR # when I get it.

UPDATE:  Sybase assigned CR 497173 to this

If you receive the 8419 error on sysusers after applying ASE 12.5.4 (or any other upgrade for that matter) such as the following:

01:00000:00472:2007/12/10 14:46:03.86 server Error: 8419, Severity: 20, State: 3
01:00000:00472:2007/12/10 14:46:03.86 server Could not find index descriptor for objid 10, indid 3 in dbid 13.
01:00000:00472:2007/12/10 14:46:03.88 kernel ************************************
01:00000:00472:2007/12/10 14:46:03.88 kernel SQL causing error : select * from sysusers
01:00000:00472:2007/12/10 14:46:03.88 kernel ************************************
01:00000:00472:2007/12/10 14:46:03.88 server SQL Text: select * from sysusers
01:00000:00472:2007/12/10 14:46:03.89 kernel curdb = 13 tempdb = 9 pstat = 0×10000
01:00000:00472:2007/12/10 14:46:03.89 kernel lasterror = 8419 preverror = 0 transtate = 1

Fear not because Drew has you covered with a simple script that will fix the problem. Note that it involves a dump and load of the affected database but you could just restart the ASE server if you don’t mind the down time:

Bau-jen Liu over at Sybase came up with the following method but it does require rebooting ASE twice:

Shutdown ASE, reboot, and log back in as a user with sa_role.

The above CREATE INDEX command may fail due to the presence of duplicate key rows in sysusers. If this occurs, recreate the index as non-unique first:

Now identify any duplicates that may exist in sysusers:

Remove any duplicates found using the DELETE command and try the CREATE INDEX again:

Now, clean up after yourself:

Sybase has created bug # 298620 for this issue.

In today’s edition of the Sybase RepServer Quirks we take a look at the “out of mutexes” error.  A mutex is an exclusive lock on a resource.  If you receive an error about RepServer running out of mutexes, you just don’t have enough defined through RepServer.  Simple huh?

Of course it is.  Ahh..  but then how do how many mutexes your particular Sybase RepServer needs?  How many grains of sand are on the moon?  According to Sybase you should know this already but since you can’t read minds (I hope you can’t because I’m thinking of donuts and not RepServer), I’ll pass on what Sybase is saying when pressed about it (Greg Carter @ Sybase):

> Just to elaborate a bit more on this question of the number of mutexes; as
> you probably know the mutex requirements for RepServer increased
> dramatically with the SMP feature. I have since struggled to come up with a
> formula for estimating mutex requirements so that you may properly set the
> “num_mutexes” configuration. While in the latest iteration of this formula
> I have satisfied myself that all mutexes have been accounted for, still the
> estimate it provides seems to fall short in some cases.
>
> Recent investigations by Connectivity seem to indicate that the problem may
> not be with sizing mutex requirements, but rather with sizing message queue
> (“num_msgqueues”) requirements. It appears that Open Server may be using
> the total of the settings for “num_mutexes” and “num_msgqueues” as the
> upper bound for the creation of these two objects together. So it may be
> that even though “num_mutexes” has been sized properly, if “num_msgqueues”
> is too low then you may see a message regarding the failure to create a
> mutex or the failure to create a message queue depending on which one was
> being created at the time that upper bound was surpassed.
>
> The moral here is that until Open Server resolves this issue you need to
> verify the sizing of both “num_mutexes” and “num_msgqueues” in the event
> that either error message appears since you can not rely on the message to
> indicate which one is low.
>
> For your convenience I’ll include here the latest formulas for estimating
> mutex and message queue requirements. Note that the one for mutexes may not
> agree completely with the one that is given in the 12.6 SMP White Paper – I
> have not compared them.
>
> Mutex requirements for the optimized binary:
> num_mutexes = 75 + Num(partitions) + 4*Num(DSI/S) + 3*Num(DSI/E) +
> 2*Num(Dist) + 2*Num(RepAgent Exec) + 2*Num(RSI User) + 5*Num(Queues) +
> 5*Num(SQT Cache) + Num(rs_subscriptions rows) + Num(RSSD tables) +
> Setting(cm_max_connections) +
> 2*MAX(Admin connections) + 2*Num(Other Connections) + Num(Origins) +
> 2*Num(Threads) + MAX(subscription (de)mat)
>
> Where
> – “Other Connections” are connection to this RepServer including ID Server
> connections, RSM connections, etc.
> – “Origins” are the different origins (or primary databases) that could
> possibly have transactions flowing through this RepServer, whether by a
> RepAgent or by a route (intermediate included)
> – “Threads” includes every thread RepServer may start. These are the
> “Global” thread, the “Initialization” thread, threads for each of the
> daemons (dAIO, dCM, dVersion, dRec, dSub, dStats, dAlarm), RepAgent User
> threads, SQM Writer threads, SQT threads, Distributor threads, DSI/S and
> DSI/E threads, RSI User and RSI threads.
> – “subscription (de)mat” is the number of asynchronous subscription
> management requests for materializing or dematerializing that may be taking
> place at any moment.
>
> Note: For the diagnostic binary you will need to double the figure
> determined with the above formula.
>
> Message queue requirements for the optimized and diagnostic binary:
> num_msgqueues = 10 + Num(DSI/S) + Num(DSI/E) + Num(Queues) + Num(Dist)
>
> Thanks,
> G.Carter

Neither this explanation or the two equations are anywhere in the manuals.  I’ve opened feature request 485482 for RepServer to handle this automagically as there really is NO NEED for a RepServer admin to have to worry about this.  If you are or ever have run into this problem, give Sybase a holler and tell them to fix this bug.

 device_fragments               size          usage                created             free kbytes
 ------------------------------ ------------- -------------------- ------------------- ----------------
 master                                8.0 MB data and log         Dec 16 2004  4:09AM             3376
 tempdev1                           1024.0 MB data and log         Dec 16 2004  7:02AM          1044480
 tempdev1                            256.0 MB data and log         Mar  9 2005 10:49AM           261120
 tempdev1                            256.0 MB log only             Mar  9 2005 10:54AM not applicable
 tempdev1                            488.0 MB data and log         Mar  9 2005 12:10PM           497760
 tempdev1                            256.0 MB data and log         Mar 11 2005  2:03PM           261120
 tempdev1                            244.0 MB data and log         Mar 11 2005  3:09PM           248880
 tempdev1                            200.0 MB data and log         Jun  1 2005 12:33PM           204000
 tempdev1                            200.0 MB data and log         Jul  8 2005  2:55PM           203808
 tempdev1                            200.0 MB data and log         Jul 21 2005  2:54PM           204000
 tempdev1                           1000.0 MB data and log         Feb  6 2006  1:11PM          1020000

So, how to fix this? Well, Sybase doesn’t provide any way to do so without modifying the system tables.

That’s it! It’s fixed and running with the segmaps without having to restart ASE. Wasn’t that easy?

1. CR 450004 – XP Server and Backupserver (64bit)
2. CR 432482 – ASE 15

The problem is the result of using the jmp() and longjmp() functions in the GNU C Library version 2.4 (glibc 2.4).

setjmp() and longjmp() are useful for dealing with errors and interrupts encountered in a low-level subroutine of a program. setjmp() saves the stack context/environment in env for later use by longjmp(). The stack context will be invalidated if the function which called setjmp() returns.

Starting in glibc 2.4, the jmpbuf is unavailable due to a security exploit known for several years.  Restricting access to the jmpbuf was an effort by the glibc maintainers to ‘close this hole’ by forcing the application writers to use the sigsetjmp() and siglongjmp() instead.

The problem with using sigsetjmp() and siglongjmp() is that they aren’t thread safe.  This is supposed to be fixed in glibc 2.6 according to a contact at Sybase but Sybase *does* have a workaround for this issue and will be putting it into the ASE v15 codeline when they perform the RHEL 5 certification sometime next year.

I wasn’t able to find much information the supposed exploit.  I only found a few newsgroup posts that didn’t provide any details.

Michael Peppler maintains the ASE on Linux FAQ.  It contains a workaround for various distributions of Linux:

Installing/Running ASE on systems with glibc 2.4.x (Fedora Core 5, etc)

ASE (12.5.x, 15.x) will not run on a system with glibc 2.4.x (that’s Fedora Core 5 and later, among others). Here is a work-around, supplied by “Chris” in sybase.public.ase.linux:

Symptoms of the problem are ASE’s failing to build when you run the installer, or “Segmentation Fault” when trying to start an ASE instance. The Seg Fault happens right after Engine ) is brought online in the error log.
If you are running Fedora Core 5 or 6
You will need to set the environment variable LD_POINTER_GUARD to 0, in your sybase .profile, as follows:

export LD_POINTER_GUARD=0

Alternatively, just export this from the shell. If you haven’t built any ASE’s yet, you’ll need to export this before running the installer, otherwise ASE will fail to build. If you are trying to boot an already installed ASE, then put it in your .profile / .bash_profile or export it as an environment variable before starting ASE.
If you are running Ubuntu / Kubuntu “Edgy”
You will need to set the environment variable LD_POINTER_GUARD to 1 in your sybase .profile, as follows:

export LD_POINTER_GUARD=1

I googled around a bit trying to find out what the LD_POINTER_GUARD thingie did and ran across a posting / blog where it talked about insome glibc versions it’s backwards … RedHat just made a change I guess and that’s why you have to set it to “0″ for Fedora and “1″ for Ubuntu / Kubuntu / Debian (glibc 2.4). Same info applies if you are building a new ASE.
If you are running Suse 10.1 You will need to set the environment variable LD_POINTER_GUARD to 1 in your sybase .profile, as follows:

export LD_POINTER_GUARD=1

Same discussion as for Ubuntu.
I’ve tried all of the above they all work with the workaround listed. Happy ASE-ing! I’ve tested all of the above distros (sorry it took me so long, was on the road a lot over the last several weeks) and they allow ASE to boot; the workaround should be valid for ASE 15.x and 12.x (prolly even for 11.x if you wanted to try).

UPDATE: Workaround for all current versions of ASE at http://froebe.net/blog/2008/06/26/how-to-install-and-run-sybase-adaptive-server-enterprise-1254-and-150x-on-ubuntu-linux-804-jeos-hardy-heron-using-chroot-environment/