I tried to compile the following code:

extern "C" {
    #include "netcdf.h"

int main() {
    const int Ntime = 336;
    const int Nlon = 1442;
    const int Nlat = 1021;
    double* dhsum_vals = new double[Ntime * Nlat * Nlon];

When compiling with the 32-bit version, I get the error C2148 "total size of array must not exceed 0x7fffffff bytes". I think, the problem here is that a 32-program does not allow to use as much memory as I want here. If I compile with the 64-bit version, I get the error, that netcdf cannot be found as I downloaded the 32-bit version. So, my solution will be to download the 64-bit version of the netcdf library. However, my intention, when taking the 32-bit version, was that it should be more compatible. Is there another solution for my problem than taking the 64-bit version of netcdf?

  • 2
    double* dhsum_vals = new double[Ntime * Nlat * Nlon]; That is not on the stack. However if this is a 32 bit program you will be limited to probably around 1.2GB for the largest allocation on windows by default because of the 2GB user space / kernel split and memory fragmentation. – drescherjm Jul 14 '20 at 15:20
  • @drescherjm, you're right, misread, and assumed stack. My mistake. – ChrisMM Jul 14 '20 at 15:23
  • 1
    The /LARGEADDRESSAWARE linker flag can help with this along with dll rebasing however it will be much simpler to just switch to 64 bit. I remember playing this game in the early 2000s but I was glad once we switched all machines to x64 so I did no longer have to deal with this. I mentioned that in this answer: https://stackoverflow.com/a/19911739/487892 – drescherjm Jul 14 '20 at 15:25
  • 1
    There is some amount of overhead ("bookkeeping") associated with using dynamic memory (malloc(), new). For a huge array, the percentage of the wasted overhead is tiny, but since you don't need it to be dynamic (do you?), you might as well just make it statically allocated, assuming you won't need multiple copies of it (e.g. one per class instance, if you're programming with C++ classes). Assuming you'll only have one such array in your whole program, you can just declare it outside of any function, and it becomes "global". – phonetagger Jul 14 '20 at 18:30
  • 1
    If you want it to be a member of a C++ class, and if you'll only need one such array regardless of the number of class instances you'll make, then you can declare it static and then there will only be one such array in the whole program, used by all instances of that class. If (going back to the previous "global array") you add the static keyword in front of the definition, then the array will still be statically allocated ("global"), but it will only be known inside that translation unit (C++ file). Note that the terms "static" and "statically allocated" are only loosly related. – phonetagger Jul 14 '20 at 18:33

On Microsoft Windows, the virtual address space of a 32-bit process is limited to 2^32 bytes, which is about 4 GB. However, the higher 2 GB are reserved by the system, so that you effectively only have about 2 GB of address space.

Also, this address space is fragmented, so you will not be able to allocate 2 GB in one consecutive chunk. You may be able to get up to 2 GB if you try several smaller memory allocations.

Although it is possible for a 32-bit process to use more than 4 GB of memory using the Address Windowing Extensions API, this requires special security privileges and makes your code unnecessarily complex, as you cannot have all the memory you are using mapped into your virtual address space at once. Therefore, I strongly suggest that you compile a 64-bit version of your application instead, when you are handling large amounts of memory.

I don't think that using 64-bit will make your program any less compatible, as nearly all modern hardware and operating systems support it.

  • I remember investigating AWE about 15 years ago. Too much effort and complexity added for what I needed. – drescherjm Jul 14 '20 at 15:55

Your Answer

