Converting 64-bit float to 32-bit float

[Tomas Gustafsson]

Task: To use .net "UdpClient" object to send byte packets:

UdpClient.SendAsync(Byte[], Int32, IPEndPoint)

The SendAsync gracefully accepts ⎕DR 83 data from APL. As is the case with all UdpClient methods, it seems. Apparently APL converts to byte data on the fly.

Problem: I must create 32-bit floats before ⎕DR 83'ing them. Since APL only knows ⎕DR 645 decimal value, i must (?) do the conversion using custom code:

      D_B4←{                                ⍝ Any number(s) -> 32-bit float(s), output as ⎕DR 83 bytes, lenght 4×⍴⍵
         a←⍴w←,⍵
         b←⍉(a,64)⍴⌽((8×a),8)⍴11 ⎕DR⊃((⎕DR w),645)⎕DR w
         83 ⎕DR,⌽((4×a),8)⍴(0≠w)∧[1]⍉(23↑29↓b)⍪(⊖(8⍴2)⊤¯896+2⊥⊖52↓¯1↓b)⍪b[64;]
     }

This is clumsy enough. Involves several ⎕DR's, transposes, rotates, takes, drops, bitwise encoding. Above all, the "endian'ess" is major trouble. I have to internally mirror all bits in groups of 8, to get it right. The Dfn originates from this experimental one:

      R←test a;b;c;d;e;expo;frac;sign
⍝ Horizontal version (has zero support, but not signed zero)
 b←⍴,a
 c←(b,64)⍴⌽((8×b),8)⍴11 ⎕DR⊃((⎕DR a),645)⎕DR,a ⍝ mirror endian
 sign←c[;64] ⍝ sign
 expo←52↓[2]¯1↓[2]c ⍝ exponent (11 bits)
 frac←23↑[2]29↓[2]c ⍝ fraction (first 29 bits dropped, 23 taken - 52 in total)
 R←83 ⎕DR,⌽((4×b),8)⍴(0≠,a)∧[1]frac,(⌽⍉(8⍴2)⊤¯896+2⊥⊖⍉expo),sign

https://en.wikipedia.org/wiki/Single-precision_floating-point_format
http://www.binaryconvert.com/convert_double.html
http://www.binaryconvert.com/convert_float.html

One could think of using ⎕OPT ("Variant") to call SendAsync with correctly cast arguments, but it's a nog go, at least in this case, fails. One could think of populating an Array object using ⎕OPT, but that also fails:

      ⍝ (#.SYS.MotionNS.array.SetValue ⎕OPT('CastToTypes'(#.Single #.Int32)))1.1 2
⍝ EXCEPTION: Cannot convert to type System.Int32

... ie. the index "2" is the problem, array lenght is 12 (it turns out the examples at http://help.dyalog.com/14.1/Content/RelNotes14.0/Specifying%20Overloads%20and%20Casts%20for%20NET%20functions.htm are somewhat special cases).

One could think of using BitConverter or System.Convert, but these are all no go's, since as soon as an intermediate result enters APL space, it magically changes itself to an APL native datatype. Tuns out there are very bad options for this in .net, at least i didn't find a single solution, after having spent quite some time on this.

Anyway... A question and a commment:

Question: What's the secret? :-) Will APL do this conversion double -> single(as byte) for me, natively? Any better way to do this?

Comment: I really think APL could have an option for "hard types and sizes", may it even be only inside a function local variable space. The ability to set these specs for a declared varaible, which no longer can hold any other data type or size. I know this would require a bit of extra housekeeping, but it would also eliminate some, or even a lot. A step closer to compiling.. Maybe you could cheat, and do the internal arithmetics by first secretly shifting to established APL types and then shift back the result?

In any case, people have had problems with the Byte format (essentially 8 or 32-bit stuff) numerous times, i believe. It would help if APL had something to make things a bit more convenient. The Variant may be good at certain situations, but native types would eliminate that one, and likely be much faster than Variant'ing on the fly (since correct types would slot in directly and correctly, kind of).

[PGilbert]

I don't have the answer to your question but in case this help I am using the following function to construct a .Net Byte[] (byte array):

Code: Select all

 ba ← ByteArray bytes;⎕IO;⎕USING
⍝ Function to construct a .Net byte array
⍝ bytes = numbers in the range of 0 to 255
⍝ ba    = .Net Byte[]

 ⎕USING ⎕IO ← 'System,mscorlib.dll' 0
 ba ← Array.CreateInstance(Byte (⍴,bytes))  ⍝ Empty array of type 'Byte'
 {ba.Set ⍵}¨↓(⍳⍴,bytes),[0.5] bytes         ⍝ Populate the Byte array


I would agree with you that we need a way to force a .Net object to stay on the .Net side. Another APL vendor is using a special ⎕ for that: ⎕REF that is append to the result. There is a comment from Morten at the end of this post related to your issue.

Regards,

Pierre Gilbert

[Tomas Gustafsson]

Thx Pierre, yes that's a natural way of starting to attack the overall problem, and i assume it works for your purposes, ie. you pass the Array "bytes" in form of 8-bit ⎕DR 82's or ⎕DR 83's. And then i guess Dyalog do a smart conversion on the fly, to let the numbers/chars slot in, ending up as bytes. My problem however was that i tried an Array of "Singles" but only had APL "doubles" (the standard APL ⎕DR 645) as the only source data format. Hence i had to try with the Variant operator ⌹ (or ⎕OPT in Classic APL) that specifies/alters the data format on the fly, on it's way into the .net Array object. However, that again failed on an error when the argument was:

1.1 2

where 1.1 is the data and 2 is the index into the array. What happens is either that APL keeps also the "2" as ⎕DR 645, as the 1.1 enforces that format for the vector. OR, as i have seen happening, Dyalog even seems to sometimes dismantle a homogenous ⎕DR 645 vector into individual APL data formats per element, on it's way into .net. So the other cause of error i can think of is that APL *did* change 1.1 into a single, but then failed on converting the "2" (that was changed into ⎕DR 83, on the fly) into the Int32 that the Array's index entry wanted. The Array call is:

SetValue(Object, Int32)

and as i wrote earlier, i tried with

(#.SYS.MotionNS.array.SetValue ⎕OPT('CastToTypes'(#.Single #.Int32)))1.1 2

I can only imagine how tricky this has been for Dyalog, to match correctly in all situations. It must be a jungle :-), so hats of for making it work this good. But i always wonder if this kind of high automation *alone* is the way to go. APL's data format is highly slippery, the advantage being ease of use as it's so highly dynamic. This is why we code 10 times faster than anyone else :-). But this kind of automation, how efficient even it is, is doomed to walk in the dark, since it has no way of knowing what the coder really wants. I have so many times said that it would be both smart and polite if the interpreter gave me a chance to tell it. Since i know what i want, and it does not.

Imagine a function header:

foo;a;b(float);c

This would tell APL that i want b to always have a format of Single, when i query the format or when i pass b to a .net function. This would only work for local variables, that would be a good start.

Then imagine these lines:

[1] a←b←c←1

Now a and c would be of whatever format the interpreter finds convenient, just as it does now. But b would be a 32-bit float.

[2] b←b+b←a+c

When performing the arithmetics, APL could well translate b into the "whateverformatAPLfindsconvenient" and do the internal calculation just like happens now. But always when it assigns b, it would go to the format specified by me. Apparently there would be some error handling needed etc., but hey something similar must be going on in there already, as APL is able to choose the good formats.

The advantage, from my "feelslike" perspective being that

1) faster execution, a step closer to compiling parts of the code; eg. if *all* involved variables in a statement were hardtyped and hardsized, maybe that statement alone could be compiled or compiledisched (hey, good idea, no? All variables in an expression typed = precompile-or-something it)?. And sure we could have a lot of such expressions, or even all, in any single function. And

2) i could have all formats set up for .net correctly in advance, and John D. wouldn't need to do all this magic in matching. I'd just pass it to any .net function, format matches, no need for Variants, much more straightforward into .net, less overhead.

[Geoff|Dyalog]

The MEMCPY routine we supply in dyalog64.so (Unix) or dyalog64.dll (Windows) is just a cover for the C libary call to memcpy().

However, the conversions Dyalog does on the way in and out of the call on MEMCPY can be very useful. So in this case

Code: Select all

      'f8tof4bytes' ⎕na'dyalog64.so|MEMCPY >U1[4] <F4 P'
      f8tof4bytes 0 1 4
0 0 128 63

Let me explain this in detail. The argument given to MEMCPY is
0 This is an output argument so this is just a placemarker to keep the arguments aligned.
1 The value 1 which will be converted to a pointer to a 32 bit float (F4) before the function is called.
4 The number of bytes to copy

The MEMCPY routine runs and copies 4 bytes from one place to the other. Now this other
place is an array of 4 unsigned 1 byte integers. Dyalog collects this result and presents the numbers.

So to go back the other way:

Code: Select all

      'f4bytestof8' ⎕na'dyalog64.so|MEMCPY >F4 <U1[4] P'
      f4bytestof8 0 (0 0 128 63) 4
1

It does not matter how the original number (1 in this case) is stored in Dyalog the representation passed to MEMCPY is specified by the F4. Provided the number can be represented as a 32 bit float it will be converted.

[Tomas Gustafsson]

Heh, thanks Geoff. The creativity of a Dyalog coder is limitless, it seems. I guess nobody said one cannot (misab)use memcpy this way :-).

But... longer numeric vectors, in parallel? It's this frame rate, that i have to maintain... in a hurry.

[Geoff|Dyalog]

Extending the conversion to an array is easy. In parallel, harder, and to be honest the marshalling of ⎕NA arguments is not the fastest piece of code in Dyalog. Convenient, flexible, powerful - yes all of those - fast - hmmm. However, it might be fast enough or at least faster than an alternative piece of Dyalog code.

Code: Select all

      'f8tof4bytes' ⎕na'dyalog64.so|MEMCPY >U1[] <F4[] P'
      ∇ convert←{
         bytes←4×≢⍵
         f8tof4bytes bytes ⍵ bytes
      }
      ∇

So the output is now an array of unspecified size. The argument supplied is the number of elements in the result.
The whole array of input numbers is converted to F4 and a pointer to the conversion passed the MEMCPY.
The last argument is the number of bytes to copy.

[Geoff|Dyalog]

As an alternative you do not need all of the extra primitives.
I used an AIX version of Dyalog

      alt←{                             
     ⎕IO←0     
     nelts←≢⍵     
    ⍝ floating point is a biased exponent and a mantissa          
    ⍝ 64 bit floats use a bias of 2*1023              
    ⍝ 32 bit floats use a bias of 2*127                  
     adjustbias←⍵×1.8928834978668395E¯270 ⍝ 2*127-1023                     
    ⍝ That will have forced the numbers to be floating point (I hope)      
    ⍝ I suspect that rounding is not required. Truncation will do.         
     bits64←(nelts 64)⍴11 ⎕DR adjustbias                                   
    ⍝ At this point we just need to index the appropriate bits according to
    ⍝ endianness of the source and desired endianness of the result        
    ⍝ life is easier in big endian. little is an exercise for the student. 
    ⍝ but it just changes the values of the indices
     bits32←bits64[;0,4+⍳31]  
     83 ⎕DR,bits32
 }   

       alt ,1.1
63 ¯116 ¯52 ¯52
       'f4bytestofloat' ⎕na'dyalog64.so|MEMCPY >F4 <I1[4] P'
      f4bytestofloat 0 (63 ¯116 ¯52 ¯52) 4   
1.099999905
      ⍝ just try 1 bit either side to see if we could have done better
       f4bytestofloat 0 (63 ¯116 ¯52 ¯53) 4
1.099999785
      f4bytestofloat 0 (63 ¯116 ¯52 ¯51) 4
1.100000024

[Geoff|Dyalog]

We supplied MEMCPY for just this sort of purpose.

The secondary use of extracting data from structures returned as pointers from "interesting" library calls came, slightly, later. Later in this case is probably just during a development cycle. I think users will have seen both uses together.

For many years, if I wanted to force a Dyalog crash I used

      ⎕NA'dyalog32|MEMCPY I4 <I4 I4'
     MEMCPY 0 0 4

Which copies 4 bytes from a valid pointer to a 4 byte zero to address 0. 0 is (nearly) always an invalid address. Today's paged systems do not map the first page.

Why I would want to force a crash is a peculiarity to a developer.

The reason it is MEMCPY and supplied as a Dyalog library is due to the Windows C runtime library not being accessible from ⎕NA on Windows. Then trying to help portability means we also supply it on Unix where accessing the system library directly is possible.

[Geoff|Dyalog]

Being my own student. The indices for little endian 64 to little endian 32 are

      bits32←bits64[;35 36 37 38 39 24 25 26 43 44 45 46 47 32 33 34 51 52 53 54 55 40 41 42 56 60 61 62 63 48 49 50]

[Tomas Gustafsson]

Good idea, to simply pre-adjust the exponent with a multiplication (though i'm a bit unsure how exponent vs. mantissa behave at certain values.

But this definitely fails on Windows APL, as you concluded. For 1.1, it would return

¯44 ¯52 ¯52 ¯52 when it should be ¯52 ¯52 ¯116 63

Swapping the endian means extracting bits, exponent and mantissa separately, and re-arraning & re-building them, in groups of 8.

(Ah, i can see now, when pre.viewing this post, that you already picked them :-))

As an alternative you do not need all of the extra primitives.
I used an AIX version of Dyalog

      alt←{                             
     ⎕IO←0     
     nelts←≢⍵     
    ⍝ floating point is a biased exponent and a mantissa          
    ⍝ 64 bit floats use a bias of 2*1023              
    ⍝ 32 bit floats use a bias of 2*127                  
     adjustbias←⍵×1.8928834978668395E¯270 ⍝ 2*127-1023                     
    ⍝ That will have forced the numbers to be floating point (I hope)      
    ⍝ I suspect that rounding is not required. Truncation will do.         
     bits64←(nelts 64)⍴11 ⎕DR adjustbias                                   
    ⍝ At this point we just need to index the appropriate bits according to
    ⍝ endianness of the source and desired endianness of the result        
    ⍝ life is easier in big endian. little is an exercise for the student. 
    ⍝ but it just changes the values of the indices
     bits32←bits64[;0,4+⍳31]  
     83 ⎕DR,bits32
 }   

       alt ,1.1
63 ¯116 ¯52 ¯52
       'f4bytestofloat' ⎕na'dyalog64.so|MEMCPY >F4 <I1[4] P'
      f4bytestofloat 0 (63 ¯116 ¯52 ¯52) 4   
1.099999905
      ⍝ just try 1 bit either side to see if we could have done better
       f4bytestofloat 0 (63 ¯116 ¯52 ¯53) 4
1.099999785
      f4bytestofloat 0 (63 ¯116 ¯52 ¯51) 4
1.100000024