size of workspace : no limit for later? I hope it

[andriambololona]

I return towards you to speak again about the available memory to work with APL because at present with machines 64 bits I hope that you work on the possibility of to give to APL to work without limit of memory (size of workspace) with machines 64 bits with which the price falls considerably. You are not without knowing it the office 2010 is there and it is capable of working with billion of lines in memory (EXCEL 2010) with machines 64 bits. At present I bought a laptop from 64 bits and a machine server in 64 bits to be able to work with on volumes of considerable data (in billion lines in a matrix if it is possible) is it a dream?
Je reviens vers vous pour parler de nouveau de la mémoire disponible pour travailler avec APL car actuellement avec les machines 64 bits j'espère que vous travailler sur la possibilité de de donner à APL de travailler sans limite de mémoire (taille espace de travail) avec les machines 64 bits dont le prix baisse considérablement. Vous n'êtes pas sans le savoir l'office 2010 est là et il est capable de travailler avec des millairs de lignes en mémoire (EXCEL 2010) avec les machines 64 bits. Actuellement j'ai acheté un laptop à 64 bits et une machine serveur à 64 bits pour pouvoir travailler avec sur des volumes de données considérables (au milliards de lignes dans une matrice si c'est possible) est-ce un rêve ?

[AndyS|Dyalog]

A workspace is a contiguous block of memory, in which there are arrays and other objects. Arrays in both 32 and 64 bit Dyalog APL are currently (as of V12.1 and prior) limited as follows:

the maximum length along any dimension of the array is (2*32)-1
the maximum length of the ravel of the array is (2*32)-1

We intend to increase these limits in forthcoming versions of Dyalog APL; that limit will not be (2*64)-1, but will still be very large - we expect that limit to be over (2*48)-1, possibly much larger.

So with 64 bit Dyalog APL you can today have very large workspaces, limited only by the amount of memory that you can get in your computer. Under Windows, there are additional limitations imposed by the way that a process space is populated. In particular, DLLs have to be loaded into that process space; if they are loaded in the middle of that process space, then the size of the workspace will be smaller than expected - it needs that contiguous block of memory. So under 64-bit Windows you still may not get as big a workspace as you might expect. This is not an issue under UNIX-based operating systems, where customers have already used workspaces in excess of 9GB.

Some observations:

Don't forget that you must allow for free workspace where the interpreter can put temporary arrays, and don't forget that it is very easy for an array to get much bigger if its data type changes. Add 0.1 to a vector of 0 and 1s, and the resulting array could require up to 64 times as much space ! This is true of all versions of Dyalog APL, but you may need to be more aware of it with huge arrays.

A graph of memory prices looks like a hockey stick; the price of DIMMs starts to rocket when you get to above 8GB.

You do not want to oversubscribe your memory; paging/swap spaces are all well and good, but think about how long it takes to read or write 1GB of data, and you can see that paging will effectively stop your process in its tracks !

If you are dealing with huge arrays, processing them takes time .. and to an end user, that might look like the application has hung.