Can a mapped_file larger than 2GB be random seeked?
(I'm resending this because it seems that my other message got garbled.) I'm writting on random positions on a mapped file, but it seems that if I seek past 2GB, write something, then seek to a lower position, I got a segfault. Here is the code: #include <cassert> #include <iostream> #include <fstream> #include <boost/iostreams/stream.hpp> #include <boost/iostreams/device/mapped_file.hpp> using namespace std; using boost::iostreams::stream; using boost::iostreams::mapped_file; int main(int argc, char** argv) { const size_t GIGA = 1024l*1024l*1024l; const size_t FILESIZE = 2l * GIGA + 10000; ofstream block("file", ios::binary | ios::out); assert(block.good()); block.seekp(FILESIZE); block.write("", 1); block.close(); mapped_file mappedFile("file"); stream<mapped_file> file(mappedFile, ios::binary|ios::out); assert(mappedFile.is_open()); char buffer[10] = {0}; cout << 0 << endl; file.seekp(0); file.write(buffer, 1); cout << FILESIZE - 1 << endl; file.seekp(FILESIZE - 1); file.write(buffer, 1); cout << 1 * GIGA << endl; file.seekp(1 * GIGA); file.write(buffer, 1); return 0; } I'm compiling under Ubuntu 14.10 (amd64): $ g++ boost-mmap.cpp -lboost_iostreams $ ./a.out 0 2147493647 Segmentation fault (core dumped) Everything is 64-bits: $ file a.out a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=d39e3dc0c99e666631747e496bf798b92e4d4c71, not stripped I have the same issue compiling with Visual Studio and running under Windows 8 (amd64). What I'm doing wrong? -- []s, Narcélio.
On Mon, May 4, 2015 at 2:15 PM, Narcélio Filho <narcelio@gmail.com> wrote:
(I'm resending this because it seems that my other message got garbled.)
I'm writting on random positions on a mapped file, but it seems that if I seek past 2GB, write something, then seek to a lower position, I got a segfault. Here is the code:
#include <cassert> #include <iostream> #include <fstream> #include <boost/iostreams/stream.hpp> #include <boost/iostreams/device/mapped_file.hpp>
using namespace std; using boost::iostreams::stream; using boost::iostreams::mapped_file;
int main(int argc, char** argv) { const size_t GIGA = 1024l*1024l*1024l; const size_t FILESIZE = 2l * GIGA + 10000;
ofstream block("file", ios::binary | ios::out);
assert(block.good());
block.seekp(FILESIZE); block.write("", 1); block.close();
mapped_file mappedFile("file"); stream<mapped_file> file(mappedFile, ios::binary|ios::out);
assert(mappedFile.is_open());
char buffer[10] = {0};
cout << 0 << endl; file.seekp(0); file.write(buffer, 1);
cout << FILESIZE - 1 << endl; file.seekp(FILESIZE - 1); file.write(buffer, 1);
cout << 1 * GIGA << endl; file.seekp(1 * GIGA); file.write(buffer, 1);
return 0; }
I'm compiling under Ubuntu 14.10 (amd64):
$ g++ boost-mmap.cpp -lboost_iostreams $ ./a.out 0 2147493647 Segmentation fault (core dumped)
Everything is 64-bits:
You are certain that 'everything' is 64-bits? Without diving too deep into any code, this sounds vaguely, suspiciously like a 32-bit addressing issue. It could even be something as 'simple' as underlying file API, or other, code using shorts, ints, instead of longs, for legacy purposes. Or layers built on said underlying API (i.e. boost wrappers).
$ file a.out a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=d39e3dc0c99e666631747e496bf798b92e4d4c71, not stripped
I have the same issue compiling with Visual Studio and running under Windows 8 (amd64).
What I'm doing wrong?
-- []s, Narcélio.
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Hi Michael!
I'm compiling under Ubuntu 14.10 (amd64):
$ g++ boost-mmap.cpp -lboost_iostreams $ ./a.out 0 2147493647 Segmentation fault (core dumped)
Everything is 64-bits:
You are certain that 'everything' is 64-bits? Without diving too deep into any code, this sounds vaguely, suspiciously like a 32-bit addressing issue. It could even be something as 'simple' as underlying file API, or other, code using shorts, ints, instead of longs, for legacy purposes. Or layers built on said underlying API (i.e. boost wrappers).
$ file a.out a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=d39e3dc0c99e666631747e496bf798b92e4d4c71, not stripped
Yes, I'm pretty sure that all my system and boost libs are all 64 bits: $ dpkg --get-selections | grep boost-iostream libboost-iostreams-dev:amd64 install libboost-iostreams1.55-dev:amd64 install libboost-iostreams1.55.0:amd64 install $ file /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.55.0 /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.55.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=a4839c7904aba3cb68a66739aa801e4d5d50c14f, stripped I'm using just boost API, that calls mmap(). Is this a boost bug? Using mmap() directly, it works as expected: #include <sys/types.h> #include <sys/stat.h> #include <unistd.h> #include <fcntl.h> #include <sys/mman.h> #include <iostream> #include <fstream> #include <cassert> using namespace std; int main(int argc, char *argv[]) { const size_t GIGA = 1024l*1024l*1024l; const size_t FILESIZE = 2l * GIGA + 10000; ofstream block("file", ios::binary | ios::out); assert(block.good()); block.seekp(FILESIZE); block.write("", 1); block.close(); auto fd = open("file", O_RDWR); if (fd == -1) { perror("Error opening file for reading"); exit(EXIT_FAILURE); } char *map = (char*)mmap(0, FILESIZE, PROT_WRITE, MAP_SHARED, fd, 0); if (map == MAP_FAILED) { close(fd); perror("Error mmapping file"); exit(EXIT_FAILURE); } char buffer[10] = {0}; cout << 0 << endl; lseek(fd, 0, 0); write(fd, buffer, 1); cout << FILESIZE - 2 << endl; lseek(fd, FILESIZE - 2, 0); write(fd, buffer, 1); cout << 1l*GIGA << endl; lseek(fd, 1l*GIGA, 0); write(fd, buffer, 1); if (munmap(map, FILESIZE) == -1) { perror("Error un-mmapping the file"); } close(fd); return 0; } $ g++ --std=c++11 ex.cpp $ ./a.out 0 2147493646 1073741824 -- []s, Narcélio.
In the original code you're seeking to FILESIZE - 1, but in this code you're seeking to FILESIZE - 2 On Tuesday, May 5, 2015, Narcélio Filho <narcelio@gmail.com> wrote:
Hi Michael!
I'm compiling under Ubuntu 14.10 (amd64):
$ g++ boost-mmap.cpp -lboost_iostreams $ ./a.out 0 2147493647 Segmentation fault (core dumped)
Everything is 64-bits:
You are certain that 'everything' is 64-bits? Without diving too deep into any code, this sounds vaguely, suspiciously like a 32-bit addressing issue. It could even be something as 'simple' as underlying file API, or other, code using shorts, ints, instead of longs, for legacy purposes. Or layers built on said underlying API (i.e. boost wrappers).
$ file a.out a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=d39e3dc0c99e666631747e496bf798b92e4d4c71, not stripped
Yes, I'm pretty sure that all my system and boost libs are all 64 bits:
$ dpkg --get-selections | grep boost-iostream libboost-iostreams-dev:amd64 install libboost-iostreams1.55-dev:amd64 install libboost-iostreams1.55.0:amd64 install
$ file /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.55.0 /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.55.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=a4839c7904aba3cb68a66739aa801e4d5d50c14f, stripped
I'm using just boost API, that calls mmap(). Is this a boost bug?
Using mmap() directly, it works as expected:
#include <sys/types.h> #include <sys/stat.h> #include <unistd.h> #include <fcntl.h> #include <sys/mman.h>
#include <iostream> #include <fstream> #include <cassert>
using namespace std;
int main(int argc, char *argv[]) { const size_t GIGA = 1024l*1024l*1024l; const size_t FILESIZE = 2l * GIGA + 10000;
ofstream block("file", ios::binary | ios::out); assert(block.good());
block.seekp(FILESIZE); block.write("", 1); block.close();
auto fd = open("file", O_RDWR);
if (fd == -1) { perror("Error opening file for reading"); exit(EXIT_FAILURE); }
char *map = (char*)mmap(0, FILESIZE, PROT_WRITE, MAP_SHARED, fd, 0);
if (map == MAP_FAILED) { close(fd); perror("Error mmapping file"); exit(EXIT_FAILURE); }
char buffer[10] = {0};
cout << 0 << endl; lseek(fd, 0, 0); write(fd, buffer, 1);
cout << FILESIZE - 2 << endl; lseek(fd, FILESIZE - 2, 0); write(fd, buffer, 1);
cout << 1l*GIGA << endl; lseek(fd, 1l*GIGA, 0); write(fd, buffer, 1);
if (munmap(map, FILESIZE) == -1) { perror("Error un-mmapping the file"); }
close(fd);
return 0; }
$ g++ --std=c++11 ex.cpp $ ./a.out 0 2147493646 1073741824
-- []s, Narcélio. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org <javascript:;> http://lists.boost.org/mailman/listinfo.cgi/boost-users
It doesn't matter. Same code doesn't work when using boost mapped_file. On 05/05/2015 04:47 AM, james wrote:
In the original code you're seeking to FILESIZE - 1, but in this code you're seeking to FILESIZE - 2
On Tuesday, May 5, 2015, Narcélio Filho <narcelio@gmail.com <mailto:narcelio@gmail.com>> wrote:
Hi Michael!
>> I'm compiling under Ubuntu 14.10 (amd64): >> >> $ g++ boost-mmap.cpp -lboost_iostreams >> $ ./a.out >> 0 >> 2147493647 >> Segmentation fault (core dumped) >> >> Everything is 64-bits:
> You are certain that 'everything' is 64-bits? Without diving too deep > into any code, this sounds vaguely, suspiciously like a 32-bit > addressing issue. It could even be something as 'simple' as underlying > file API, or other, code using shorts, ints, instead of longs, for > legacy purposes. Or layers built on said underlying API (i.e. boost > wrappers). > >> $ file a.out >> a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically >> linked (uses shared libs), for GNU/Linux 2.6.32, >> BuildID[sha1]=d39e3dc0c99e666631747e496bf798b92e4d4c71, not stripped
Yes, I'm pretty sure that all my system and boost libs are all 64 bits:
$ dpkg --get-selections | grep boost-iostream libboost-iostreams-dev:amd64 install libboost-iostreams1.55-dev:amd64 install libboost-iostreams1.55.0:amd64 install
$ file /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.55.0 /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.55.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=a4839c7904aba3cb68a66739aa801e4d5d50c14f, stripped
I'm using just boost API, that calls mmap(). Is this a boost bug?
Using mmap() directly, it works as expected:
#include <sys/types.h> #include <sys/stat.h> #include <unistd.h> #include <fcntl.h> #include <sys/mman.h>
#include <iostream> #include <fstream> #include <cassert>
using namespace std;
int main(int argc, char *argv[]) { const size_t GIGA = 1024l*1024l*1024l; const size_t FILESIZE = 2l * GIGA + 10000;
ofstream block("file", ios::binary | ios::out); assert(block.good());
block.seekp(FILESIZE); block.write("", 1); block.close();
auto fd = open("file", O_RDWR);
if (fd == -1) { perror("Error opening file for reading"); exit(EXIT_FAILURE); }
char *map = (char*)mmap(0, FILESIZE, PROT_WRITE, MAP_SHARED, fd, 0);
if (map == MAP_FAILED) { close(fd); perror("Error mmapping file"); exit(EXIT_FAILURE); }
char buffer[10] = {0};
cout << 0 << endl; lseek(fd, 0, 0); write(fd, buffer, 1);
cout << FILESIZE - 2 << endl; lseek(fd, FILESIZE - 2, 0); write(fd, buffer, 1);
cout << 1l*GIGA << endl; lseek(fd, 1l*GIGA, 0); write(fd, buffer, 1);
if (munmap(map, FILESIZE) == -1) { perror("Error un-mmapping the file"); }
close(fd);
return 0; }
$ g++ --std=c++11 ex.cpp $ ./a.out 0 2147493646 1073741824
-- []s, Narcélio. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org <javascript:;> http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- []s, Narcélio.
participants (4)
-
james -
Michael Powell -
Narcélio Filho