variable definition in used file cause huge run time hit

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

variable definition in used file cause huge run time hit

adrianv
I have been puzzling over a run time optimization issue where I have code
that calls a select() function to do indexing.  The select function gets
called many times (thousands).   I run "use <math.scad>" where math.scad
contains:

function select(vector,start,end) = [ for (index = [start:end])
vector[index] ];

function ident(n) = [for (i = [0:n-1]) [for (j = [0:n-1]) (i==j)?1:0]];
ident3 = ident(3);   // This line
ident4 = ident(4);  // and this line cause huge slow down

My code with many calls to select runs in 19s.  But if I remove the two
assignments to ident3 and ident4 the run time falls to 6s.  What is going on
here?  

I tried using include instead of use and then there was no penalty for
having assignments in the included file.  In fact, the code ran a bit
faster: 5s.  (I can't tell if the difference between 5s and 6s is real.)   I
tried adding definitions for ident5 and ident6 just to see what would
happen.  Witih "use", run time went to 9 minutes 11 s.  Yikes.  That's quite
a performance hit for something that could have been done in 5s.  

So why does this happen?  It's not a fixed overhead.  If I decrease my
number of calls to select() then the code with the offending lines included
runs in 0s.  Inserting an echo() into ident() reveals that ident() is not
actually being run (unless the output of echo is somehow supressed).  So
what is going on here?  

Suppose I actually needed definitions like ident3 and ident4 in my code.  Is
there a way to do them that won't create this performance hit for unrelated
functions in file?   I had imagined that they get computed once and are then
available, so this would be more efficient than defining them within a
function that used them.  Is this a mistaken assumption?  





--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: variable definition in used file cause huge run time hit

nophead
Oh wow this seems like a huge problem. I put x = echo(42) in one of my used modules and found it echos every time I use a function or a module from that file! 

Not sure why you didn't see an echo in ident() but I think select slows down because every time you run select it is computing ident3 and ident4 as well.

On Sun, 31 Mar 2019 at 05:42, adrianv <[hidden email]> wrote:
I have been puzzling over a run time optimization issue where I have code
that calls a select() function to do indexing.  The select function gets
called many times (thousands).   I run "use <math.scad>" where math.scad
contains:

function select(vector,start,end) = [ for (index = [start:end])
vector[index] ];

function ident(n) = [for (i = [0:n-1]) [for (j = [0:n-1]) (i==j)?1:0]];
ident3 = ident(3);   // This line
ident4 = ident(4);  // and this line cause huge slow down

My code with many calls to select runs in 19s.  But if I remove the two
assignments to ident3 and ident4 the run time falls to 6s.  What is going on
here?   

I tried using include instead of use and then there was no penalty for
having assignments in the included file.  In fact, the code ran a bit
faster: 5s.  (I can't tell if the difference between 5s and 6s is real.)   I
tried adding definitions for ident5 and ident6 just to see what would
happen.  Witih "use", run time went to 9 minutes 11 s.  Yikes.  That's quite
a performance hit for something that could have been done in 5s. 

So why does this happen?  It's not a fixed overhead.  If I decrease my
number of calls to select() then the code with the offending lines included
runs in 0s.  Inserting an echo() into ident() reveals that ident() is not
actually being run (unless the output of echo is somehow supressed).  So
what is going on here? 

Suppose I actually needed definitions like ident3 and ident4 in my code.  Is
there a way to do them that won't create this performance hit for unrelated
functions in file?   I had imagined that they get computed once and are then
available, so this would be more efficient than defining them within a
function that used them.  Is this a mistaken assumption? 





--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: variable definition in used file cause huge run time hit

RevarBat
Wow.  I've confirmed this behavior in OS X OpenSCAD 2019.03.03

- Revar


On Mar 31, 2019, at 1:03 AM, nop head <[hidden email]> wrote:

Oh wow this seems like a huge problem. I put x = echo(42) in one of my used modules and found it echos every time I use a function or a module from that file! 

Not sure why you didn't see an echo in ident() but I think select slows down because every time you run select it is computing ident3 and ident4 as well.

On Sun, 31 Mar 2019 at 05:42, adrianv <[hidden email]> wrote:
I have been puzzling over a run time optimization issue where I have code
that calls a select() function to do indexing.  The select function gets
called many times (thousands).   I run "use <math.scad>" where math.scad
contains:

function select(vector,start,end) = [ for (index = [start:end])
vector[index] ];

function ident(n) = [for (i = [0:n-1]) [for (j = [0:n-1]) (i==j)?1:0]];
ident3 = ident(3);   // This line
ident4 = ident(4);  // and this line cause huge slow down

My code with many calls to select runs in 19s.  But if I remove the two
assignments to ident3 and ident4 the run time falls to 6s.  What is going on
here?   

I tried using include instead of use and then there was no penalty for
having assignments in the included file.  In fact, the code ran a bit
faster: 5s.  (I can't tell if the difference between 5s and 6s is real.)   I
tried adding definitions for ident5 and ident6 just to see what would
happen.  Witih "use", run time went to 9 minutes 11 s.  Yikes.  That's quite
a performance hit for something that could have been done in 5s. 

So why does this happen?  It's not a fixed overhead.  If I decrease my
number of calls to select() then the code with the offending lines included
runs in 0s.  Inserting an echo() into ident() reveals that ident() is not
actually being run (unless the output of echo is somehow supressed).  So
what is going on here? 

Suppose I actually needed definitions like ident3 and ident4 in my code.  Is
there a way to do them that won't create this performance hit for unrelated
functions in file?   I had imagined that they get computed once and are then
available, so this would be more efficient than defining them within a
function that used them.  Is this a mistaken assumption? 





--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: variable definition in used file cause huge run time hit

tp3
See https://github.com/openscad/openscad/issues/782

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: variable definition in used file cause huge run time hit

nophead
Amazing, nearly 5 years old.

I have a few global definitions in my library that are included in every library file I use. I thought that was about 70 times but adding an echo I realise it more like 5000 times because every time I call an external module or function they get defined again.

I also have some projects where a used file might have a lot of moderately complex variable definitions that use functions. For example calculating screw lengths. If the file exports a few functions and modules itself they get repeated many times when it is used only once in another file.

I don't know if I rely on this behaviour because my mental model of how OpenSCAD scope works is broken by this. How easy is it to change to see what it breaks and what the speed increase is?

On Sun, 31 Mar 2019 at 12:59, Torsten Paul <[hidden email]> wrote:
See https://github.com/openscad/openscad/issues/782

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: variable definition in used file cause huge run time hit

adrianv
I do not understand why the echo() I inserted into ident() wasn't printing a
lot yesterday.  Today it is.  

This seems like a catastrophic problem that should be at the top of the list
to be fixed.  The example in the issue for a possible use case is contrived,
and the behavior we have now is strongly surprising.  If we are going to
keep it this way intentionally we need to document the heck out of it---and
give a solid justification for use cases---but I would say we should fix it
and whatever it breaks (if anything) needs to deal with the fix.  Basically
if a library needs nontrivial local variables you can't use "use" but must
use "include".   Using "include" means the namespace is cluttered (my
expensive library variables don't need to be exposed) and it means a user
needs to remember to use "include" sometimes.  This leads to the notion that
"use" is dangerous and can't be used: just always use "include".  

The issue was posted in 2014 and the statement was made that it was "easy to
fix".   Is it so?  




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: variable definition in used file cause huge run time hit

tp3
On 31.03.19 15:31, adrianv wrote:
> This seems like a catastrophic problem that should be at the top
> of the list to be fixed.

I'd suggest to calm down a bit. We should not be in any kind of
life or death situation, even if OpenSCAD has helped saving lifes
in the past
(https://hackaday.com/2018/03/17/3d-printed-stethoscope-makes-the-grade/).

> If we are going to keep it this way intentionally we need to
> document the heck out of it

We will do none of that.

It's a bug and it should be fixed. However we are talking spare
time of a small number of volunteers, so there is no such thing
as a time line. There might be ways to motivate people to care
about a special priority, but shouting end-of-the-world is
probably not helping.

> The issue was posted in 2014 and the statement was made that
> it was "easy to fix".   Is it so?

No idea, it sounds like Oskar did understand the fix at that
time, so hopefully it's true. But we will only know for sure
after it's done.

ciao
   Torsten.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten