Question about the new ⎕JSON coming up with v16

General APL language issues

Question about the new ⎕JSON coming up with v16

Postby PGilbert on Wed Dec 07, 2016 12:46 am

We are using the Dyalog JSON script to serialize/de-serialize Apl data to text file like this: [2,2,3,1,2,3,4,5,6] ←→ 2 3⍴⍳6 . The script is rather slow, will the new ⎕JSON will be able to do what the Dyalog JSON script is doing. In the negative why not ? Is there another way to serialize/de-serialize apl data to text file that is quick and supported by Dyalog ?

Thanks in advance,

Pierre Gilbert
User avatar
PGilbert
 
Posts: 401
Joined: Sun Dec 13, 2009 8:46 pm
Location: Montréal, Québec, Canada

Re: Question about the new ⎕JSON coming up with v16

Postby Phil Last on Wed Dec 07, 2016 11:51 am

Hey Pierre
This will do the job for simple data or down to depth two:
      encode←7160⌶∘{(⍴⍴⍵)(⍴⍵)(,⍵)}
- substitute ⎕JSON for 7160⌶ where available - but requires a doubly recursive call for stuff more deeply nested
      encode{⍺⍺ ⍺⍺¨⍵}
There will certainly be other edge cases that cause problems but this'll do most of what you want.
There are other problems with 7160⌶ concerned with empties. It encodes both and 0⍴⊂⍬ as the same empty list [] This doesn't overcome that.
Controlling when to use recursion and decode are left to the reader.
User avatar
Phil Last
 
Posts: 557
Joined: Thu Jun 18, 2009 6:29 pm

Re: Question about the new ⎕JSON coming up with v16

Postby Phil Last on Wed Dec 07, 2016 12:20 pm

Of course you will have realised that as the control data are prefixed without catenation we don't actually need to include the rank.
      encode←7160⌶∘{(⍴⍵)(,⍵)} ⋄ decode←{⊃⍴/7159⌶⍵}
User avatar
Phil Last
 
Posts: 557
Joined: Thu Jun 18, 2009 6:29 pm

Re: Question about the new ⎕JSON coming up with v16

Postby Morten|Dyalog on Sun Dec 11, 2016 12:19 pm

v16 will include ⎕JSON as a replacement for the experimental I-Beams (which will be retained for one release in order to allow for an orderly transition). The option to add metadata and convert APL arrays into something that can be represented in JSON was provided by the JSON utility library, but is not included in the new definition of ⎕JSON (nor was it supported by the I-Beams).

In an e-mail, you sent me some data to play with: The only problem you have is that some of your variables are matrices: JSON only supports vectors. I experimented with the following alternatives for serialisation, which are sufficient if you only have ranks greater than 1 at the top level.

As we have already seen, one strategy is to add shape information to each array and ravel the data:

Code: Select all
     ∇ r←toJSON2 ns;vars;ns2
[1]    'ns2'⎕NS ⎕OR'ns' ⍝ clone
[2]    vars←⍕ns2.⎕NL-2
[3]    ns2⍎'(',vars,')←{(⍴⍵) (,⍵)}¨',vars
[4]    r←⎕JSON ns2
     ∇
     ∇ r←fromJSON2 json;vars
[1]    r←⎕JSON json
[2]    vars←⍕r.⎕NL-2
[3]    r⍎'(',vars,')←⊃¨⍴/¨',vars
     ∇

Second alternative: Add an extra variable called METADATA, containing the shapes (and ravel the data):

Code: Select all
     ∇ r←toJSON3 ns;vars;ns2
[1]    'ns2'⎕NS ⎕OR'ns' ⍝ clone
[2]    ns2.(METADATA←{⍵(⍴¨⍎⍕⍵)}⎕NL ¯2)
[3]    vars←⍕1⊃ns2.METADATA
[4]    ns2⍎vars,'←,¨',vars
[5]    r←⎕JSON ns2
     ∇
     ∇ r←fromJSON3 json;vars;shapes
[1]    r←⎕JSON json
[2]    vars←⍕1⊃r.METADATA
[3]    r⍎'(',vars,')←(2⊃METADATA)⍴¨',vars
[4]    r.⎕EX'METADATA'
     ∇

The performance of the alternatives is almost identical. On my laptop, the old JSON namespace functions use 180ms + 230ms to convert your from APL to JSON and back (the JSON representation uses about 130kb to represent your 33 variables). The new functions use 15ms + 5ms; just over 10x faster for the serialisation and nearly 50x faster when reconstructing the namespace. The extra lines of code don't seem to add much to the time spend by the system function.

A final point worth making: If you want to serialise data as JSON, you might want to consider using data structures which are all directly supported (in other words, no matrices). Then you will just be able to use ⎕JSON directly, with the added benefit that non-APL code will be able to understand the data too, without additional special knowledge.
User avatar
Morten|Dyalog
 
Posts: 394
Joined: Tue Sep 09, 2008 3:52 pm

Re: Question about the new ⎕JSON coming up with v16

Postby PGilbert on Sun Dec 11, 2016 4:37 pm

Thanks Morten and Phil, this is answering my question and Wow the code is extremely fast! For the benefit of everybody here is the functions that I was looking for:

Code: Select all
 json←SimpleAplToJSON apl;vars;apl2
⍝ Converts Simple APL Variable and Nameless Namespace of Simple APL Variables to JSON
⍝ Works with the function 'JSONtoSimpleApl' to Deserialize the JSON back to APL data
⍝ Code taken from: http://www.dyalog.com/forum/viewtopic.php?f=13&t=1118

⍝ Examples:
⍝ ⍳5      ←→ [[5],[1,2,3,4,5]]
⍝ 2 5⍴⍳10 ←→ [[2,5],[1,2,3,4,5,6,7,8,9,10]]

⍝ ns ← ⎕ns''
⍝ ns.data1 ← ⍳5
⍝ ns.data2 ← 2 5⍴⍳10
⍝ ns.data3 ← 'This is some text'
⍝ ns ←→ {"data1":[[5],[1,2,3,4,5]],"data2":[[2,5],[1,2,3,4,5,6,7,8,9,10]],"data3":[[17],"This is some text"]}

 :If 2=⎕NC'apl'
    ⍝ Simple APL variable
     json←7160⌶{(⍴⍵)(,⍵)}apl

 :ElseIf 9=⎕NC'apl'
    ⍝ Nameless Namespace of Simple APL Variables
     'apl2'⎕NS ⎕OR'apl' ⍝ clone
     vars←⍕apl2.⎕NL-2
     apl2⍎'(',vars,')←{(⍴⍵) (,⍵)}¨',vars
     json←7160⌶apl2

 :Else
    ⍝ Don't know what to do
     json←⎕NULL

 :EndIf

Code: Select all
 apl←JSONtoSimpleApl json;vars
⍝ Converts JSON to Simple APL Variable or Nameless Namespace of Simple APL Variables
⍝ Works with the function 'SimpleAplToJSON' to Serialize the APL data to JSON
⍝ Code taken from: http://www.dyalog.com/forum/viewtopic.php?f=13&t=1118
⍝ See 'SimpleAplToJSON' for examples

 :If '['=1↑json
    ⍝ Simple APL variable
     apl←⊃⍴/7159⌶json

 :ElseIf '{'=1↑json
    ⍝ Nameless Namespace of Simple APL Variables
     apl←7159⌶json
     vars←⍕apl.⎕NL-2
     apl⍎'(',vars,')←⊃¨⍴/¨',vars

 :Else
     apl←⎕NULL

 :EndIf
User avatar
PGilbert
 
Posts: 401
Joined: Sun Dec 13, 2009 8:46 pm
Location: Montréal, Québec, Canada


Return to Language

Who is online

Users browsing this forum: No registered users and 1 guest