O_DIRECT & O_ASYNC, Linux & Perl

I’ve written a C program that creates a 20GB file using O_DIRECT and O_ASYNC correctly. The trick to write to a file using O_DIRECT is that I have to align the buffer with respect to the memory block size.

When I write a similar program using Perl, I receive the expected “System write error: Invalid argument” as the buffer is not aligned.

So, my question is: Is it possible, in Perl, to align the buffer?

#!/usr/bin/perl

use strict;
use warnings;

$|++;

use Fcntl qw(:DEFAULT O_ASYNC O_DIRECT);

my $FH;
sysopen($FH, "./test.dat", O_WRONLY | O_TRUNC | O_CREAT | O_ASYNC | O_
+DIRECT, 0666);

my $BUFFER = "0"x1048576;
my $BUFSIZE = 1048576;

for (my $i = 0; $i < 20480; $i++) {
my $written = syswrite($FH, $BUFFER, $BUFSIZE);
die "System write error: $!\\n" unless defined $written;
}

Don’t forget Mother’s Day! It’s this Sunday so get her some sock yarn.

Update: sgifford over at Perlmonks.org, gave me the answer. Just mmap() the buffer using Sys::Mmap 🙂

#!/usr/bin/perl

use strict;
use warnings;

$|++;

use Fcntl qw(:DEFAULT O_ASYNC O_DIRECT);
use Sys::Mmap;

my $FH;
sysopen($FH, "./test.dat", O_WRONLY | O_TRUNC | O_CREAT | O_ASYNC | O_DIRECT, 0666);

my $BUFSIZE = 1048576;
my $BUFFER;

mmap($BUFFER, $BUFSIZE, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, STDOUT) or die "Couldn't mmap\\n";

for (my $i = 0; $i < 20480; $i++) {
        my $written = syswrite($FH, $BUFFER, $BUFSIZE);
        die "System write error: $!\\n" unless defined $written;
}
Share Button

ASE 15.0.1 esd 2 on Linux with direct i/o woes

If you’re having trouble creating a device file on Linux while using direct io, you’re not alone:

disk init name = data01, 
physname = "/dbms/sybase/data/data01.dat", 
size = "20G", 
dsync = false, 
directio = true
03:00000:00041:2007/05/04 21:20:20.21 kernel Initializing virtual device 4, '/dbms/sybase/data/data01.dat' with dsync 'off'.
03:00000:00041:2007/05/04 21:20:20.21 kernel Virtual device 4 started using asynchronous (with O_DIRECT) i/o.
03:00000:00041:2007/05/04 21:20:20.21 kernel Initializing device /dbms/sybase/data/data01.dat from offset 0 with zeros.
00:00000:00000:2007/05/04 21:21:59.76 kernel engine 1, os pid 28232 exited
00:00000:00000:2007/05/04 21:22:09.85 kernel Process 28232 exited with status 0
00:00000:00000:2007/05/04 21:22:09.85 kernel Engine exited from scheduler, lastkpid b000b
00:00000:00000:2007/05/04 21:22:09.85 kernel Engine exited with signal 9
00:00000:00000:2007/05/04 21:22:09.94 kernel engine 3, os pid 28234 exited
00:00000:00000:2007/05/04 21:22:09.94 kernel Process 28234 exited with status 0
00:00000:00000:2007/05/04 21:22:09.94 kernel Task with kpid 260026 running on failed engine
00:00000:00000:2007/05/04 21:22:09.94 kernel Engine exited with signal 9
02:00000:00000:2007/05/04 21:22:15.50 kernel kesetown: engine 1 not online
02:00000:00000:2007/05/04 21:22:15.50 kernel kesetown: engine 3 not online
02:00000:00020:2007/05/04 21:22:17.63 server Restarted process 'HK GC'.
00:00000:00000:2007/05/04 21:22:17.63 kernel kpenginedied: killed kpid 655370 which was affinitied to engine 3
00:00000:00000:2007/05/04 21:22:17.63 kernel kpenginedied: killed kpid 1703962 which was affinitied to engine 3
00:00000:00000:2007/05/04 21:22:17.63 kernel kpenginedied: killed kpid 1769499 which was affinitied to engine 3
00:00000:00000:2007/05/04 21:22:17.63 kernel kpenginedied: killed kpid 1835036 which was affinitied to engine 3
00:00000:00000:2007/05/04 21:22:17.63 kernel kpenginedied: killed kpid 2031647 which was affinitied to engine 3
00:00000:00000:2007/05/04 21:22:17.63 kernel kpenginedied: killed kpid 2162721 which was affinitied to engine 3

The odd thing is that 17GB device files work fine but 17.5 GB and larger device files fail on the disk init. Even stranger, the device is created (for device file sizes 17.5GB < -> 20GB – any larger and the device isn’t available on restart) but you need to restart ASE for it to be available.

Workaround:

Create device file without direct i/o and then use sp_deviceattr to add the direct i/o flag. Note you will still need to restart for ASE to open the device files with direct i/o.

I have an open case with Sybase so we will have a CR (bug) filed shortly.

I’m running into this on Red Hat Enterprise Linux AS release 4 (Nahant Update 5). I haven’t attempted this on any of the other distros.

UPDATE: This appears to be specific to RHEL 4 release 4 update 5. Sybase is having trouble reproducing it but I can reproduce it on two machines. Go figure. I’ll provide more updates as we make progress.

UPDATE: I created a C program that creates a 20GB file using O_DIRECT & O_ASYNC with no problems. I’m suspecting that ASE is doing something funky with the kernel threads that might be causing this problem.

Share Button