Intuition, aesthetics and performance
15 posts
• Page 1 of 2 • 1, 2
Intuition, aesthetics and performance
Two equivalent algorithms:
⊃{⍺,'%',⍵}/⍵
1↓⊃,/'%',¨⍵
Both chain a list of strings with intervening separators (represented here by '%').
I don't get to examine other people's code much but several of us are working on a system that's currently about 10MB in about 5000 individual functions, 100 operators, 300 namespaces and 75 classes.
There are 76 occurrences of the former and 8 of the latter.
You can see why. I didn't really need to explain what they did because the first one says so on the tin. It's nice to use self documenting algorithms and we all know code clarity is important for maintenance as well as pleasing to the eye.
Unfortunately:
At 10 strings the nice one takes twice as long.
At 100 strings it's 5 times.
At 1000, 100.
At 10000, my comparison function gets a domain error equivalent to divide by zero.
There is hope; the former is also equivalent to:
⊃'%'{⍺,⍺⍺,⍵}/⍵
which is marginally slower (10%) than the first at less than 10 strings and equivalent above 100. But its advantage is that it's invariant from the left brace rightwards.
In other words {⍺,⍺⍺,⍵}/ might be a candidate for a recognised idiom.
⊃{⍺,'%',⍵}/⍵
1↓⊃,/'%',¨⍵
Both chain a list of strings with intervening separators (represented here by '%').
I don't get to examine other people's code much but several of us are working on a system that's currently about 10MB in about 5000 individual functions, 100 operators, 300 namespaces and 75 classes.
There are 76 occurrences of the former and 8 of the latter.
You can see why. I didn't really need to explain what they did because the first one says so on the tin. It's nice to use self documenting algorithms and we all know code clarity is important for maintenance as well as pleasing to the eye.
Unfortunately:
At 10 strings the nice one takes twice as long.
At 100 strings it's 5 times.
At 1000, 100.
At 10000, my comparison function gets a domain error equivalent to divide by zero.
There is hope; the former is also equivalent to:
⊃'%'{⍺,⍺⍺,⍵}/⍵
which is marginally slower (10%) than the first at less than 10 strings and equivalent above 100. But its advantage is that it's invariant from the left brace rightwards.
In other words {⍺,⍺⍺,⍵}/ might be a candidate for a recognised idiom.
-
Phil Last - Posts: 628
- Joined: Thu Jun 18, 2009 6:29 pm
- Location: Wessex
Re: Intuition, aesthetics and performance
Why don't you refactor all of the repeated code into a function? Call it "join". That way it can be ugly if it needs to for speed (and it's less characters to type).
We have exactly such a function and it used to look just like your first example but we re-wrote it into the second form since it was so much quicker (quadratic performance is really bad for something simple like concatenation) The re-write was easy because the definition was in one place, the join function.
While idioms are nice I would much rather have general optimisations that apply everywhere, like say reducing interpreter overhead or adding some sort of loop fusion to reduce memory traffic.
We have exactly such a function and it used to look just like your first example but we re-wrote it into the second form since it was so much quicker (quadratic performance is really bad for something simple like concatenation) The re-write was easy because the definition was in one place, the join function.
While idioms are nice I would much rather have general optimisations that apply everywhere, like say reducing interpreter overhead or adding some sort of loop fusion to reduce memory traffic.
- harsman
- Posts: 27
- Joined: Thu Nov 26, 2009 12:21 pm
Re: Intuition, aesthetics and performance
Exactly the approach I should have taken in the past. But in present circumstances it would have to be at the least "##.Core.join" if I wanted to ensure its availability wherever used unless it was cloned into the twenty different namespaces that use it.
-
Phil Last - Posts: 628
- Joined: Thu Jun 18, 2009 6:29 pm
- Location: Wessex
Re: Intuition, aesthetics and performance
Ah, too bad. This is why I want a []PATH with namespace scope, less unnecessary typing without naming conflicts. You could make []PATH search "outwards" for all namespaces, but that might cause other issues depending on how they're structured, and it doesn't help you with sibling in the namespace tree.
- harsman
- Posts: 27
- Joined: Thu Nov 26, 2009 12:21 pm
Re: Intuition, aesthetics and performance
I consider the introduction of []PATH one of the biggest mistakes ever made in APL. For a start, it doesn't avoid naming conflicts, it makes them more effective. You can't double-click a name, or press <Shift+Enter> on it. With []PATH clarity and readability is sacrificed in order to avoid a bit of typing. We have Autocompletion to save typing time.
-
kai - Posts: 137
- Joined: Thu Jun 18, 2009 5:10 pm
- Location: Hillesheim / Germany
Re: Intuition, aesthetics and performance
harsman wrote:While idioms are nice I would much rather have general optimisations that apply everywhere, like say reducing interpreter overhead or adding some sort of loop fusion to reduce memory traffic.
This year, we will start a research project into new, more general optimizations and "just-in-time-compilation" techniques. With luck, this will eventually lead to significant reductions in overhead in various parts of the interpreter, loop fusion and significantly more efficient use of memory - and allow us to distribute work efficiently across multiple cores. It will be some time before the results of this work make it into the "main stream", but we expect the impact to be "significant".
---
Regarding ⎕PATH, do you think it would solve your problem if ⎕USING was extended to allow it to refer to APL namespaces?
-
Morten|Dyalog - Posts: 453
- Joined: Tue Sep 09, 2008 3:52 pm
Re: Intuition, aesthetics and performance
Re. ⎕PATH I agree with Kai. I'd go further. Although I've learned to live with it I consider :Include very much in the same light as ⎕PATH and other obfuscations. I didn't mention it above but my preference so far is to define a shortcut ref at the top of a function or even globally in the ns that points to the real source.
f00←{
ut←##.Core.Utils
...
z←ut.whatever ⍵
⍝ even renaming remote functions locally if I need to use them frequently
join←ut.join
x←', 'join list
...
}
f00←{
ut←##.Core.Utils
...
z←ut.whatever ⍵
⍝ even renaming remote functions locally if I need to use them frequently
join←ut.join
x←', 'join list
...
}
-
Phil Last - Posts: 628
- Joined: Thu Jun 18, 2009 6:29 pm
- Location: Wessex
Re: Intuition, aesthetics and performance
Well, with my duty to present an alternative point of view...
I'm happy to use ⎕PATH when it suits me, taking the view that I have a computer to handle menial jobs like remembering where I put things. The only real beef I have is that it's a simple character vector, I'd have thought it more "APLey" if it had been a nested vector. There are plenty more evil things in this world than ⎕PATH, even in APL.
I've used Phil's "abbreviation" approach occasionally (mostly because fully qualified names are just so tedious, not just to type but also to read) - works (for me) in simple cases but I think it breaks down if there are too many namespaces involved (just looked at a little "messing around" workspace and seen it contain a dozen "application" namespaces and thirty "utility" - can't abbreviate all of them).
Maybe I'm just lucky (surely not well-disciplined), but I don't ever seem to run into name clashes.
I'm happy to use ⎕PATH when it suits me, taking the view that I have a computer to handle menial jobs like remembering where I put things. The only real beef I have is that it's a simple character vector, I'd have thought it more "APLey" if it had been a nested vector. There are plenty more evil things in this world than ⎕PATH, even in APL.
I've used Phil's "abbreviation" approach occasionally (mostly because fully qualified names are just so tedious, not just to type but also to read) - works (for me) in simple cases but I think it breaks down if there are too many namespaces involved (just looked at a little "messing around" workspace and seen it contain a dozen "application" namespaces and thirty "utility" - can't abbreviate all of them).
Maybe I'm just lucky (surely not well-disciplined), but I don't ever seem to run into name clashes.
Visit http://apl.dickbowman.com to read more from Dick Bowman
-
Dick Bowman - Posts: 235
- Joined: Thu Jun 18, 2009 4:55 pm
Re: Intuition, aesthetics and performance
I don't really care about []PATH one way or the other, I just want a somewhat sane way to handle modularization and naming conflicts. If []PATH has namespace scope I could build a reasonable facsimile of a module system without re-writing most of our code base, but I'd be open to alternatives.
What I would like:
[]USING working for APL-namespaces or []PATH having namespace scope would solve most of these and I could build a module system on top of it, by having some magic function called import that did all the dirty work.
This is essentially how our current module system works, except it pre-dates namespaces so it is based on prefixing everything with the module name, and there is a huge pile of "utility" functions that are available in the global namespace.
What I would like:
- A standard way to say "this module/namespace uses and depends on these other modules".
A choice on whether to refer to these other modules I depend on with their full prefix (core.vec.join), something shorter I invented myself (v.join) or without any prefix at all (join).
A standard way to map namespaces/modules to the file system to enable loading them dynamically from text files or work spaces.
A pony.
[]USING working for APL-namespaces or []PATH having namespace scope would solve most of these and I could build a module system on top of it, by having some magic function called import that did all the dirty work.
This is essentially how our current module system works, except it pre-dates namespaces so it is based on prefixing everything with the module name, and there is a huge pile of "utility" functions that are available in the global namespace.
- harsman
- Posts: 27
- Joined: Thu Nov 26, 2009 12:21 pm
Re: Intuition, aesthetics and performance
Morten|Dyalog wrote:harsman wrote:While idioms are nice I would much rather have general optimisations that apply everywhere, like say reducing interpreter overhead or adding some sort of loop fusion to reduce memory traffic.
This year, we will start a research project into new, more general optimizations and "just-in-time-compilation" techniques. With luck, this will eventually lead to significant reductions in overhead in various parts of the interpreter, loop fusion and significantly more efficient use of memory - and allow us to distribute work efficiently across multiple cores. It will be some time before the results of this work make it into the "main stream", but we expect the impact to be "significant".
That sounds awesome. Have you looked at LuaJIT? It has very impressive numerical performance, especially for a dynamically typed language like Lua. I think the semantics for numbers in Lua match APL pretty well, everything can be promoted to doubles.
- harsman
- Posts: 27
- Joined: Thu Nov 26, 2009 12:21 pm
15 posts
• Page 1 of 2 • 1, 2
Who is online
Users browsing this forum: No registered users and 1 guest
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group