windows sysopen() fails to open file larger than 2**32 bytes

[p5pRT]

Migrated from rt.perl.org#113864 (status was 'open')

Searchable as RT113864$

[p5pRT]

From villek@matrixscience.com

This is a bug report for perl from villek@​matrixscience.com,
generated with the help of perlbug 1.39 running under perl 5.14.2.

---

When I try to sysopen() a file larger than 2**32 bytes on
Windows 7 x64, using ActivePerl 5.14.2 (64-bit), I get an "Invalid
argument" error. I can open() the same file without problems.

The same problem persists on Windows Server 2003 and Server 2008
(using ActivePerl 5.14.2 -- both 32-bit and 64-bit -- and ActivePerl
5.12.4 (64-bit)), as well as on Windows 7 x64 using Strawberry Perl
5.14.2 (64-bit). It is for this reason that it seems to be a bug in
Perl core itself.

It's easy to reproduce (the first command creates an empty file of
the appropriate size -- ensure you have at least 4 GB free disk
space)​:

C​:\Users\villek>perl -e "open(my $fh, '>', 'test.file') or die $!;
  truncate($fh, 2**32+1) or die $!"

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR
  | O_CREAT) or die $!; print -s $fh"
Invalid argument at -e line 1.

C​:\Users\villek>perl -e "open(my $fh, '+>>', 'test.file') or die $!;
  print -s $fh"
4294967297

Expected result​: the second and third command should open the file
successfully and print the same file size (2**32 + 1 = 4294967297).

Perl Info

-----------------------------------------------------------------
---
Flags:
category=core
severity=medium
---
Site configuration information for perl 5.14.2:

Configured by sshd_server at Fri Oct 7 15:14:49 2011.

Summary of my perl5 (revision 5 version 14 subversion 2) configuration:

Platform:
osname=MSWin32, osvers=5.2, archname=MSWin32-x64-multi-thread
uname=''
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -Ox -GL
-fp:precise -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE
-DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO',
optimize='-MD -Zi -DNDEBUG -Ox -GL -fp:precise',
cppflags='-DWIN32'
ccversion='14.00.40310.41', gccversion='', gccosandvers=''
intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8
ivtype='__int64', ivsize=8, nvtype='double', nvsize=8,
Off_t='__int64', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf
-ltcg -libpath:"C:\Perl64\lib\CORE" -machine:AMD64'
libpth=\lib
libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib bufferoverflowU.lib msvcrt.lib
perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib
winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
version.lib odbc32.lib odbccp32.lib comctl32.lib bufferoverflowU.lib
msvcrt.lib
libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl514.lib
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug
-opt:ref,icf -ltcg -libpath:"C:\Perl64\lib\CORE" -machine:AMD64'

Locally applied patches:
ACTIVEPERL_LOCAL_PATCHES_ENTRY

---
@INC for perl 5.14.2:
C:/Perl64/site/lib
C:/Perl64/lib
.

---
Environment for perl 5.14.2:
HOME (unset)
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)

PATH=C:\Perl64\site\bin;C:\Perl64\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
PERL_BADLANG (unset)
SHELL (unset)

[p5pRT]

From villek@matrixscience.com

I have run the test case on the following Windows platforms and
ActivePerl versions​:

XP Pro x64 SP2, ActivePerl 5.14.2 (64-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x64 SP2, ActivePerl 5.10.1 (64-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.14.2 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.12.4 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.10.1 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.8.9 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

Server 2003 R2 SP2 x64, ActivePerl 5.14.2 (32-bit)
- 2**32+1​: sysopen succeeds, but prints file size of 0
- 2**32+999,456,769​: same
- open always succeeds

Server 2003 R2 SP2 x64, ActivePerl 5.10.1 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

Server 2003 R2 SP2 x64, ActivePerl 5.8.8 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

Server 2008 Standard SP1 x32, ActivePerl 5.12.4 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.14.2 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.12.4 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.12.4 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.10.1 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.10.1 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

Linux versions are fine​:

Debian 6.0.5 x64, Perl 5.10.1 (64-bit)
- 2**32+1​: succeeds

CentOS 4.6 x64, Perl 5.14.0 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

CentOS 4.6 x64, Perl 5.12.0 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

CentOS 4.6 x64, Perl 5.10.0 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

CentOS 4.6 x64, Perl 5.8.7 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

Strangely, in production code on Windows Server 2008, sysopen() fails
opening a file of size 4,429,498,960 bytes (as expected from above).
However, sysopen() succeeds for a file of size 4,347,902,043 bytes.
Reason for this behaviour is unknown.

[p5pRT]

From [Unknown Contact. See original ticket]

I have run the test case on the following Windows platforms and
ActivePerl versions​:

XP Pro x64 SP2, ActivePerl 5.14.2 (64-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x64 SP2, ActivePerl 5.10.1 (64-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.14.2 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.12.4 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.10.1 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

XP Pro x32 SP3, ActivePerl 5.8.9 (32-bit)
- 2**32+1​: sysopen dies
- open always succeeds

Server 2003 R2 SP2 x64, ActivePerl 5.14.2 (32-bit)
- 2**32+1​: sysopen succeeds, but prints file size of 0
- 2**32+999,456,769​: same
- open always succeeds

Server 2003 R2 SP2 x64, ActivePerl 5.10.1 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

Server 2003 R2 SP2 x64, ActivePerl 5.8.8 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

Server 2008 Standard SP1 x32, ActivePerl 5.12.4 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.14.2 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.12.4 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.12.4 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.10.1 (32-bit)
- 2**32+1​: sysopen dies
- open succeeds

7 Ultimate x64, ActivePerl 5.10.1 (64-bit)
- 2**32+1​: sysopen dies
- open succeeds

Linux versions are fine​:

Debian 6.0.5 x64, Perl 5.10.1 (64-bit)
- 2**32+1​: succeeds

CentOS 4.6 x64, Perl 5.14.0 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

CentOS 4.6 x64, Perl 5.12.0 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

CentOS 4.6 x64, Perl 5.10.0 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

CentOS 4.6 x64, Perl 5.8.7 (64-bit)
- 2**32+1​: succeeds (up to 2**36)

Strangely, in production code on Windows Server 2008, sysopen() fails
opening a file of size 4,429,498,960 bytes (as expected from above).
However, sysopen() succeeds for a file of size 4,347,902,043 bytes.
Reason for this behaviour is unknown.

[p5pRT]

villek@matrixscience.com - Status changed from 'new' to 'open'

[p5pRT]

From tchrist@perl.com

What open flags are you using? Is this a platform that
requires O_LARGEFILES or'd into the flag arg?

--tom

[p5pRT]

From villek@matrixscience.com

On Fri Jun 29 06​:58​:30 2012, tom christiansen wrote​:

What open flags are you using? Is this a platform that
requires O_LARGEFILES or'd into the flag arg?

I retried with a few different sets of flags on Windows 7 x64, ActivePerl 5.14.2 (64-bit)
(test.file is 2**32+1 bytes)​:

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDONLY) or die $!; print -s
$fh"
4294967297

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDONLY | O_CREAT) or die $!;
print -s $fh"
4294967297

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR) or die $!; print -s
$fh"
Invalid argument at -e line 1.

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR | O_CREAT) or die $!;
print -s $fh"
Invalid argument at -e line 1.

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_WRONLY) or die $!; print -s
$fh"
4294967297

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_WRONLY | O_CREAT) or die $!;
print -s $fh"
4294967297

So apparently it's restricted to only O_RDWR -- I suppose that's good news.

O_LARGEFILE isn't available​:

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR | O_LARGEFILE) or die
$!; print -s $fh"
Your vendor has not defined Fcntl macro O_LARGEFILE, used at -e line 1

Regards,
Ville Koskinen

[p5pRT]

From villek@matrixscience.com

I should also point out that everything works fine when test.file is
2**32-1 bytes or smaller​:

C​:\Users\villek>perl -e "open(my $fh, '+<', 'test.file') or die $!;
truncate($fh, 2**32-1) or die $!"

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR) or
die $!; print -s $fh"
4294967295

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR |
O_CREAT) or die $!; print -s $fh"
4294967295

And in fact there is something entirely different happening when the
file size is exactly 2**32 bytes!

C​:\Users\villek>perl -e "open(my $fh, '+<', 'test.file') or die $!;
truncate($fh, 2**32) or die $!"

C​:\Users\villek>dir test.file
Directory of C​:\Users\villek

02/07/2012 10​:10 4,294,967,296 test.file
  1 File(s) 4,294,967,296 bytes
  0 Dir(s) 21,202,997,248 bytes free

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR) or
die $!; print -s $fh"
No such file or directory at -e line 1.

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR |
O_CREAT) or die $!; print -s $fh"
No such file or directory at -e line 1.

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDONLY)
or die $!; print -s $fh"
4294967296

Regards,
Ville Koskinen

[p5pRT]

From @bulk88

On Mon Jul 02 02​:12​:57 2012, villek wrote​:

I should also point out that everything works fine when test.file is
2**32-1 bytes or smaller​:

C​:\Users\villek>perl -e "open(my $fh, '+<', 'test.file') or die $!;
truncate($fh, 2**32-1) or die $!"

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR) or
die $!; print -s $fh"
4294967295

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR |
O_CREAT) or die $!; print -s $fh"
4294967295

And in fact there is something entirely different happening when the
file size is exactly 2**32 bytes!

C​:\Users\villek>perl -e "open(my $fh, '+<', 'test.file') or die $!;
truncate($fh, 2**32) or die $!"

C​:\Users\villek>dir test.file
Directory of C​:\Users\villek

02/07/2012 10​:10 4,294,967,296 test.file
1 File(s) 4,294,967,296 bytes
0 Dir(s) 21,202,997,248 bytes free

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR) or
die $!; print -s $fh"
No such file or directory at -e line 1.

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDWR |
O_CREAT) or die $!; print -s $fh"
No such file or directory at -e line 1.

C​:\Users\villek>perl -MFcntl -e "sysopen(my $fh, 'test.file', O_RDONLY)
or die $!; print -s $fh"
4294967296

Regards,
Ville Koskinen

Here is what I got with a x64 Perl with msvcr90.dll version 9.00.30729.1
on Server 2003 x64. Callstack params in perl517.dll are correct due to
-Od build, params in msvcr90.dll are garbage due to -O1 build by MS.
____________________________________________________________

msvcr90.dll!_lseek_nolock(int fh=1241920, long pos=258, int mthd=258)
Line 112 + 0xb bytes C
  msvcr90.dll!_tsopen_nolock(int * punlock_flag=0x000000000012f344, int
* pfh=0x0000000000000000, const char * path=0x00000000003472d8, int
oflag=27968512, int shflag=64, int pmode=438, int bSecure=0) Line 506 +
0xd bytes C
  msvcr90.dll!_open(const char * path=0x000000000034a878, int oflag=258,
...) Line 120 + 0x27 bytes C
  perl517.dll!win32_open(const char * path=0x0000000001aac4e8, int
flag=258, ...) Line 3242 C
  perl517.dll!PerlLIOOpen3(IPerlLIO * piPerl=0x000000000034a7a8, const
char * filename=0x0000000001aac4e8, int oflag=258, int pmode=438)
Line 1027 C++
  perl517.dll!PerlIOUnix_open(interpreter * my_perl=0x000000000034aff8,
_PerlIO_funcs * self=0x00000000282375c0, PerlIO_list_s *
layers=0x0000000001aac708, __int64 n=0, const char *
mode=0x000000000012f5d9, int fd=-1, int imode=258, int perm=438, _PerlIO
* * f=0x0000000000000000, int narg=1, sv * * args=0x000000000012f618)
Line 2716 + 0x33 bytes C
  perl517.dll!PerlIOBuf_open(interpreter * my_perl=0x000000000034aff8,
_PerlIO_funcs * self=0x0000000028237940, PerlIO_list_s *
layers=0x0000000001aac708, __int64 n=1, const char *
mode=0x000000000012f5d8, int fd=-1, int imode=258, int perm=438, _PerlIO
* * f=0x0000000000000000, int narg=1, sv * * args=0x000000000012f618)
Line 3857 + 0x7c bytes C
  perl517.dll!PerlIO_openn(interpreter * my_perl=0x000000000034aff8,
const char * layers=0x0000000000000000, const char *
mode=0x000000000012f5d8, int fd=-1, int imode=258, int perm=438, _PerlIO
* * f=0x0000000000000000, int narg=1, sv * * args=0x000000000012f618)
Line 1648 + 0x72 bytes C
  perl517.dll!Perl_do_openn(interpreter * my_perl=0x000000000034aff8, gv
* gv=0x0000000001ab6958, const char * oname=0x0000000001b80d08, long
len=9, int as_raw=1, int rawmode=258, int rawperm=438, _PerlIO * *
supplied_fp=0x0000000000000000, sv * * svp=0x000000000012f618, long
num_svs=1) Line 189 + 0x54 bytes C
  perl517.dll!Perl_pp_sysopen(interpreter * my_perl=0x000000000034aff8)
Line 1591 + 0x51 bytes C
  perl517.dll!Perl_runops_standard(interpreter *
my_perl=0x000000000034aff8) Line 42 + 0xd bytes C
  perl517.dll!S_run_body(interpreter * my_perl=0x000000000034aff8, long
oldscope=1) Line 2425 C
  perl517.dll!perl_run(interpreter * my_perl=0x000000000034aff8) Line
2342 C
  perl517.dll!RunPerl(int argc=2, char * * argv=0x0000000000344dc0, char
* * env=0x00000000003481c0) Line 270 + 0xa bytes C++
  perl.exe!main(int argc=2, char * * argv=0x0000000000344dc0, char * *
env=0x0000000000345140) Line 24 C
  perl.exe!__tmainCRTStartup() Line 582 + 0x19 bytes C
  kernel32.dll!BaseProcessStart() + 0x2c bytes
____________________________________________________________

The script run is
___________________________________________________________
#!/usr/bin/perl -w
use strict;
use Fcntl;
use Win32​::API;

my $db = Win32​::API->new('kernel32.dll', 'DebugBreak', '', '');
$db->Call();
sysopen(my $fh, 'test.file', O_RDWR | O_CREAT) or die $!;
print -s $fh
__________________________________________________________

I will not copy paste the CRT code here. In _tsopen_nolock a CreateFileA
is done and new handle is created successfully. Due to this open() being
with FTEXT being on (
https://subversion.assembla.com/svn/mojos/kernel/crt/src/open.c ), later
there is a comment that a CTRL-Z has to be stripped off if the file is
in FTEXT and some other conditionals. A _lseek_nolock() is called. That
calls SetFilePointer(). SetFilePointer fails with
"ERROR_INVALID_PARAMETER/87 (0x57)/The parameter is incorrect", which
maps to EINVAL/0x16. The open() returns EINVAL eventually to the interp.

Args to SetFilePointer are
HANDLE hFile = 0x000000000000038C
LONG lDistanceToMove = 0x00000000FFFFFFFF (actually 0xFFFFFFFF since
unused high bits are ignored on Win64 and LONG is 32 bits)
PLONG lpDistanceToMoveHigh = 0x0000000000000000 NULL
DWORD dwMoveMethod = 0x0000000000000002 (ignore high bits, DWORD is
32bit) = FILE_END

SetFilePointer returns 0x00000000FFFFFFFF (actually 0xFFFFFFFF), which
is INVALID_SET_FILE_POINTER with GetLastError() returning
"ERROR_INVALID_PARAMETER/87 (0x57)/The parameter is incorrect" . Per
http​://msdn.microsoft.com/en-us/library/windows/desktop/aa365541%28v=vs.85%29.aspx
, what I guess happened was, you try to open a bigger than 2^32 file,
for that lpDistanceToMoveHigh can not be NULL per SetFilePointer, since
return value and lpDistanceToMoveHigh are combined to get low 32 bits
and high 32 bits of a larger than 2^32 file. Windows won't let you
overflow per SetFilePointer's docs.

So it is either a bug or limitation in MS CRT's open(). I haven't done
any googling to see how common this problem, and if anyone else has run
into this open() with text mode on a 4GB file problem.

In the past the way Win32 Perl has dealt with CRT bugs is to
re-implement parts of the CRT (see 9b1f181 and 72e6b64). Whether this is the correct way or a way or the wrong way to fix this bug IDK.

--
bulk88 ~ bulk88 at hotmail.com

[toddr]

Can anyone with windows say if this works on newer windows with a file > 2**32 bytes?