In-Memory data compression

General APL language issues

In-Memory data compression

Postby orlandod on Fri Feb 18, 2011 4:55 pm

I am playing around with the idea of compressing cached data in order to speed up some of the processes that I run which receive multiple requests. I am storing the results in the workspace where I can then do a lookup. If the data is kept in-memory, it starts utilizing too much memory and becomes a drag on the system but it returns a response much quicker. Does anyone have any recommendations/functions for compressing/decompressing in-memory data (integers, doubles of vectors, arrays)?

Thank you,

Orlando
orlandod
 
Posts: 4
Joined: Tue Mar 09, 2010 9:15 pm

Re: In-Memory data compression

Postby orlandod on Fri Feb 18, 2011 5:01 pm

Searching on Google led me to this article:

http://www.vector.org.uk/archive/v181/hack181.htm

Thanks to Stefano and Adrian
orlandod
 
Posts: 4
Joined: Tue Mar 09, 2010 9:15 pm

Re: In-Memory data compression

Postby StefanoLanzavecchia on Mon Feb 21, 2011 8:59 am

Hi! I am the Stefano who originally wrote the code to interface ZLIB with Dyalog APL and I hope you find it useful. But if you don't, you can also check http://www.dyalog.dk/dfnsdws/n_Data_compression.htm. Have fun :)
User avatar
StefanoLanzavecchia
 
Posts: 93
Joined: Fri Oct 03, 2008 9:37 am

Re: In-Memory data compression

Postby AndyS|Dyalog on Mon Feb 21, 2011 6:50 pm

Better would be http://www.dyalog.com/dfnsdws/n_Data_compression.htm .. one day we will disable links to dyalog.dk.

The "See also" link to pack on the above page may be of interest .. rather than attempting to compress data, pack attempts to reduce the amount of workspace used by sharing sub-arrays, although that is very much data dependent.
User avatar
AndyS|Dyalog
 
Posts: 243
Joined: Tue May 12, 2009 6:06 pm


Return to Language

Who is online

Users browsing this forum: No registered users and 1 guest