Proposal: APL Language extensions

[neeraj]

As we all know we live in the array world. So everything that we need or use we arrayize. For e.g. []XML returns an array from essentially a tree structured text file. I wonder if the lanuage should / could be extended so that we can deal with other collections as arrays which are not necessarily square or rectangle for e.g. a tree. How about associative arrays or dictionaries.

Just throwing it out into the open to see what the users of this forum think.

[StefanoLanzavecchia]

Here are a couple of classes that implement an associative array (a general one and one specialised for case-insensitive strings as keys). There are probably references to utility functions that anybody should be able to easily substitute with their private versions.

Code: Select all

:Class Dict
    :field public Instance _Keys←⍬
    :field public Instance _Values←⍬

    :field public Instance _ErrorIfNotFound←0
    :field public Instance _Default←''

    :property Keyed Default Item
    :Access Public Instance
        ∇ v←get arg;k
          k←⊃arg.Indexers
          v←GetString k
          v←{(⊂⍣((0≠⊃⍴⍴v)∧0=⊃⍴⍴k))⍵}v
        ∇
        ∇ set arg;v;k
          v k←arg.(NewValue Indexers)
      ⍝    k←{(⊂⍣(2≠≡⍵))⍵}k
          (⊃k){Put ⍺ ⍵}¨v
        ∇
    :endProperty

    ∇ Put(k v);i
      i←_Keys⍳⊂k←NormalizeKey k
      :If (⍴_Keys)<i
          _Keys,←⊂k ⋄ _Values,←⊂''
      :EndIf
      (i⊃_Values)←v
    ∇

    ∇ v←GetString k
      :Access Public Instance
      k←{(⊂⍣(2≠≡⍵))⍵}k
      v←(_Values,_ErrorIfNotFound↓⊂_Default)[_Keys⍳NormalizeKey¨k]
      v←{(⊃⍣(0=⊃⍴⍴k))⍵}v
    ∇

    ∇ v←GetStringWithDefault(k def)
      :Access Public Instance
      k←{(⊂⍣(2≠≡⍵))⍵}k
      v←(_Values,⊂def)[_Keys⍳NormalizeKey¨k]
      v←{(⊃⍣(0=⊃⍴⍴k))⍵}v
    ∇

    ∇ v←GetNumber k
      :Access public instance
      :If ''≢v←GetString k
          v←⊃(//)⎕VFI v
      :EndIf
    ∇

    ∇ key←NormalizeKey key
      :Access Public Overridable
    ∇

    ∇ tbl←FromMatrix mx
      :Access Public Shared
      tbl←⎕NEW ⎕THIS
      tbl.(_Keys _Values)←↓[1]mx
    ∇


    ∇ RemoveValue v
      :Access Public Instance
      _Keys _Values⌿⍨←⊂v∘≢¨_Values
    ∇

    ∇ r←ToMatrix
      :Access Public Instance
      r←_Keys,[1.5]_Values
    ∇

:endclass

:Class DictCI : Dict
 ⍝ Dictionary with Case Insensitive keys
   
    ∇ key←NormalizeKey key
      :Access Public Override
      key←#.UPPERCASE key
    ∇
:EndClass


No documentation.
But a quick sample could be as follows. Don't expect to be able to run it... it's a snippet from production code so there's stuff missing around, but it should give a taste for the syntax.

Code: Select all

 vars←'@PDFPages'(⍕PDF.CountPages path,p)
 vars⍪←'@NomeLotto'fname

 vars⍪←'@XML'(fname,'_I.xml')
 vars⍪←'@INF'(fname,'.inf')
 vars⍪←'@CPX'(fname,'.cpx')

 vars⍪←'@Path'path

 tbl←Dict.FromMatrix vars
 tbl._Default←'*UNKNOWN*'

 dt←0 0 2000+3↑2⊃'/'⎕VFI⊃tbl[⊂'data fine reg']     ⍝ 30/11/06
 tbl[⊂'MeseRiferimento']←dt[2]
 tbl[⊂'AnnoRiferimento']←dt[3]

 attr←⎕NEW Dict
 attr[⊂'Producer']←⊂'Utente'
 attr['Id_lotto' 'Id_file' 'Info_filename' 'Prn_filename']←tbl['@NomeLotto' '@XML' '@INF' '@CPX']
 attr[⊂'Language']←⊂'PDF'


[alexbalako]

Do not forget that base language already has "nested" arrays, which may represent tree structure.

[Phil Last]

Do not forget that base language already has "nested" arrays, which may represent tree structure.

And "ordinary" namespaces allow one to implement dictionaries but without having to quote all the targets.

dict←⎕NS''
dict.(this that and the other)←Tom Dick and a Harry

[neeraj]

Thanks to everbody who replied. I would like to draw attention to the fact that []XML has been implemented as returning an array. I am not suggesting even for a moment that this is wrong in any way, just to the fact that we did not have a data structure which would be more appropriate hence it was implemented to return a rectangular array.

While the language is rich enough to provide workarounds to achieve similar effects, the purpose of my email was to point to where the APL language is deficient. It could be argued that everything that Dfns can do can be done by explicit functions so there is no need for Dfns, but look at the elegance imparted to code by DFns.

[JohnS|Dyalog]

I believe that Ken Iverson's original notation in his book "A Programming Language" did indeed have trees.

It seems to me that there is some merit in our current position that "everything is an array". Otherwise, we would have to define the effect of applying each of our primitive functions to the new data types. This would be possible but it would increase the size of the language significantly.

I wonder also, if some algebraic reasoning and expression transformation is possible only if we know that the arguments are arrays?

It's amusing to compare our predicament with that of some of the FP languages, where "everything is a list": we implement lists using arrays and they implement arrays using lists :-).

[Morten|Dyalog]

... So everything that we need or use we arrayize. For e.g. ⎕XML returns an array from essentially a tree structured text file. I wonder if the lanuage should / could be extended so that we can deal with other collections as arrays which are not necessarily square or rectangle for e.g. a tree. How about associative arrays or dictionaries.

I don't know whether I like the idea of adding data structures to APL which are not "arrays", but arrays already support at least two kinds of trees: nested arrays and nested objects.

I think it *is* worth thinking about whether we could add an operator or two which would allow the application of APL functions to nodes in a the trees that we can already represent.

We'll take a close look at this in the months to come and see whether any elegant ideas come to the surface.

Morten

P,S, With respect to ⎕XML, can you explain about some operations which you would find easier to express if the data was represented as a tree?

[neeraj]

I was exposed to []XML for the first time at the APL conference and the first thought that crossed my mind was that xml is inherently a tree like structure and is it the case that

1. Are we using an array because that is what we know

or

2. An array is the best solution even if we had a tree structure.

I have not worked sufficiently with XML that I can give you a cogent example right now where a tree structure would definitely benefit (other forum members who have more experience, please contribute). But my feeling is that we have stayed with arrays for almost 50 years now. It may be time to explore new frontiers.

While we are at it I think we should also look at text processing facilities in Dyalog natively. I think when dealing with more than a few megabytes the search facility used to become painfully slow. A solution exists in the form of using .NET external code but I believe that there are a sufficiently large no. of apps out there that would be written or benefit from superior text processing natively in APL. Once again these are my opinions and stuff may have changed since I last explored. To be fair the datasets that I have dealt with range from 100s of megabytes to terabyte size.

[AndyS|Dyalog]

That thought had occurred to us :-)

Some people wish to be able to do string matching on APL code, and others are concerned more with parsing on text files - so the trick is to come up with a solution that works with all versions of Dyalog APL, irrespective of platform, and meets as many requirements as possible. The discussions so far have been based on using PCRE-compliant functionality, allowing both pattern matching and string replacement ..

[DanB|Dyalog]

There is already a user command (named WSLOC) in 12.1 - which is V11 compatible via Spice - that allows to search/replace the ws using regular expressions. It uses .Net but is similar to PCRE.