"use" results in 3x more run time than "include"

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

nophead
Yes but I have recently removed most of the renders. I now only have them where there is a 3D difference or intersection. And I also rewrote everything to avoid as many of those as possible by using unions of linear_extrudes, rotate_extrudes and hulls.

The only render in this picture is for the solder bucket pins that are hollow tubes with a 45 degree slice off the end. And I render one pin and duplicate it rather than rendering all the gold pins together, which would have done a CGAL union.

 In the GUI this previews in 0 seconds. On the command line it takes 2 seconds.

image.png

Previously I rendered everything at the top level for each colour and this test took 44 seconds, one of the slowest. I only realised recently that OpenCSG union is totally free and that objects made from the union of lots of little bits still draw quickly. Difference and intersection take significant time and don't scale well. If I don't render them in a scene with hundreds of objects then the GUI becomes unresponsive and I get z-fighting between negative objects and they go wrong when the camera gets close.

So now I only have 14 renders in about 46 vitamin files and that is what takes most of the remaining time. These PCBs take about 4.7 seconds.

pcbs.png

The only renders left are the blue and green terminal housings that are 3D differences.

image.png

I used to do all the ways together but to speed it up greatly I render a single way and duplicate it.

Looking it at it now I realise I could decompose the connector housings into the union of lots of tiny pieces to avoid any differences and shave some more time off. Looks like it would be about 0.5 seconds.

On Mon, 15 Apr 2019 at 00:54, MichaelAtOz <[hidden email]> wrote:
Noting, for others, that nophead uses render(), so cmd-line preview includes
the CGAL processing.



-----
Admin - email* me if you need anything, or if I've done something stupid...

* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.

The TPP is no simple “trade agreement.”   Fight it! http://www.ourfairdeal.org/   time is running out!
--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

nophead
Yes each housing cell is now a union of three linear extrudes, one horizontal and two vertical, avoiding any renders. Time is now down to 4.1 seconds.

image.png
image.png
image.png


image.png



On Mon, 15 Apr 2019 at 08:29, nop head <[hidden email]> wrote:
Yes but I have recently removed most of the renders. I now only have them where there is a 3D difference or intersection. And I also rewrote everything to avoid as many of those as possible by using unions of linear_extrudes, rotate_extrudes and hulls.

The only render in this picture is for the solder bucket pins that are hollow tubes with a 45 degree slice off the end. And I render one pin and duplicate it rather than rendering all the gold pins together, which would have done a CGAL union.

 In the GUI this previews in 0 seconds. On the command line it takes 2 seconds.

image.png

Previously I rendered everything at the top level for each colour and this test took 44 seconds, one of the slowest. I only realised recently that OpenCSG union is totally free and that objects made from the union of lots of little bits still draw quickly. Difference and intersection take significant time and don't scale well. If I don't render them in a scene with hundreds of objects then the GUI becomes unresponsive and I get z-fighting between negative objects and they go wrong when the camera gets close.

So now I only have 14 renders in about 46 vitamin files and that is what takes most of the remaining time. These PCBs take about 4.7 seconds.

pcbs.png

The only renders left are the blue and green terminal housings that are 3D differences.

image.png

I used to do all the ways together but to speed it up greatly I render a single way and duplicate it.

Looking it at it now I realise I could decompose the connector housings into the union of lots of tiny pieces to avoid any differences and shave some more time off. Looks like it would be about 0.5 seconds.

On Mon, 15 Apr 2019 at 00:54, MichaelAtOz <[hidden email]> wrote:
Noting, for others, that nophead uses render(), so cmd-line preview includes
the CGAL processing.



-----
Admin - email* me if you need anything, or if I've done something stupid...

* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.

The TPP is no simple “trade agreement.”   Fight it! http://www.ourfairdeal.org/   time is running out!
--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

jon_bondy
What you are doing is very interesting and useful, but I am a little
confused.   I thought I heard that unions were time consuming, and yet
you describe doing a union of linear extrudes. Perhaps I was wrong about
thinking that unions are time consuming.

On 4/15/2019 8:55 AM, nop head wrote:
> Yes each housing cell is now a union of three linear extrudes, one
> horizontal and two vertical, avoiding any renders. Time is now down to
> 4.1 seconds.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

nophead
CGAL 3D union is time very consuming but OpenCSG union seems to have very little or no overheat at all. It simply draws the pieces on top of each other I think.

On the other hand OpenCSG difference seems to be somewhat slower because all the negative objects are drawn as well. Although they should be invisible they z-fight with other negative and positive objects they happen to line up with, giving artefacts in assembly views and if they are oversized they get clipped if you move the camera close. So I put render around 3D differences to force CGAL to compute the mesh so OpenSCG can just draw it. That is slow the first time I preview and cached after that. 

By changing as much as I can to be unions of extrudes I avoid 3D differences and can remove nearly all of my renders and get a massive speed up of the first time preview. For example my half designed 3D printer goes from about 12 minutes to 2.5 minutes. Most of that time will be rendering the non-library printed parts specific to that project. I haven't gone through all those decomposing them as it isn't worth the time for non-library parts.

When coding with this style thrown together view is no different to preview because I have removed all the negative objects from OpenCSG.

On Mon, 15 Apr 2019 at 14:06, jon <[hidden email]> wrote:
What you are doing is very interesting and useful, but I am a little
confused.   I thought I heard that unions were time consuming, and yet
you describe doing a union of linear extrudes. Perhaps I was wrong about
thinking that unions are time consuming.

On 4/15/2019 8:55 AM, nop head wrote:
> Yes each housing cell is now a union of three linear extrudes, one
> horizontal and two vertical, avoiding any renders. Time is now down to
> 4.1 seconds.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

adrianv
In reply to this post by kintel
kintel wrote
>> On Apr 14, 2019, at 03:44, nop head &lt;

> nop.head@

> &gt; wrote:
>>
>> Yes variables seem to re evaluated when you call a function in a used
>> module but I am sure functions and modules are not recompiled because
>> that is done at parse time and the AST is cached in the module cache. So
>> there isn't anything to do for a function unless it is called by a
>> variable initialisation.
>>
> I  don't think that's necessary, and even if we continue doing it, it
> should be possible to perform some sort of Common Expression Elimination.
> This probably went largely unnoticed since until the recently introduced
> echo() function, we didn't really have functions with side effects.

It didn't seem like this entirely went un-noticed, given that there's an
open issue from 2014 about it.  But without echo it's hard to really nail
down what's happening.   It seems more that people just haven't been paying
much attention to performance.  And also, perhaps due to the lack of
interest in libraries, people haven't been really trying to write basic
libraries where you might actually call library functions thousands of
times.  If your library is something that creates geometry and it takes 10
seconds to run then it doesn't much matter if there's a 0.1 s overhead
because of duplicate to level assignment execution.  

There are two things I've found that make "use" slower and I don't know if
they are manifestations of the same thing.  When I raised this topic this
time, it was not about assignments at the top level, but about (apparently)
function calls.  The first performance concern, which I noted a couple weeks
ago, is that with "use", assignments at the top level get run any time you
execute a function in the library.  This means that if you have a bunch of
assignments, perhaps to demonstrate your library, they will get executed
every time any function in the library is called from outside the library.
In my case this was giving me a run time of 10.5 minutes which I reduced to
5 seconds by commenting out the examples in the library file.  Note that
this is the specific example given for "use"---that you can include examples
and they won't be run.  

The second issue is that it appears to be relatively expensive to call a
function from a "use"ed library if that library is large.  I have a test
case where I have the following simple function

function quant(x,y) = floor(x/y+0.5)*y;

in math.scad, a large library file (1500 lines of code).  If I run this:

use<BOSL/math.scad>
function quantvect(vec,scale) = [for (v=vec) quant(v,scale)];  

x = rands(1,1000,1e5);
y=(quantvect(x,.01));

the run time is 12s.  It does not matter whether quant() is defined first or
last in math.scad---run time is the same.  If I change "use" to "include"
run time is 0s.  If I instead put quant() in a library all by itself and
"use" then run time is 0s.  And of course if I put the definition of quant()
directly into my code run time is 0s.  

I don't know what, exactly, is causing the slow down but it has to do with
having 10^5 function calls into the large library.  If I put quantvect()
into math.scad so there is only 1 function call to the library, it runs in
0s.  

It does appear that the obvious solution is to simply ignore "use" and
always use "include".  Is there some disadvantage to this approach?  I
suppose you lose the ability to "use" files from a library and not expose
their contents to the to level "user".  Nophead reported a slight
slowdown...do to reparsing?  I'm not sure I understand why that would
happen.  




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

boxcarmib
> It does appear that the obvious solution is to simply ignore "use" and
> always use "include".  Is there some disadvantage to this approach?  I

I use “use” over include in that it allows me to keep test invocations in library files. Include as you know will execute any 0-level commands it encounters when being included… where as “use” only loads the function/module definitions and executes nothing.


> suppose you lose the ability to "use" files from a library and not expose
> their contents to the to level "user".  Nophead reported a slight
> slowdown...do to reparsing?  I'm not sure I understand why that would
> happen.  
>
>
>
>
> --
> Sent from: http://forum.openscad.org/
>
> _______________________________________________
> OpenSCAD mailing list
> [hidden email]
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

acwest
I believe "include" doesn't properly track relative directories when including other files, while "use" does, although this may have been resolved at some point. I haven't checked lately

On Tue, 16 Apr 2019, 20:12 Hugo Jackson, <[hidden email]> wrote:
> It does appear that the obvious solution is to simply ignore "use" and
> always use "include".  Is there some disadvantage to this approach?  I

I use “use” over include in that it allows me to keep test invocations in library files. Include as you know will execute any 0-level commands it encounters when being included… where as “use” only loads the function/module definitions and executes nothing.


> suppose you lose the ability to "use" files from a library and not expose
> their contents to the to level "user".  Nophead reported a slight
> slowdown...do to reparsing?  I'm not sure I understand why that would
> happen. 
>
>
>
>
> --
> Sent from: http://forum.openscad.org/
>
> _______________________________________________
> OpenSCAD mailing list
> [hidden email]
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

nophead
The reason l haven't noticed is that l simply don't have thousands of function calls across file boundaries. My projects typically spend a second or two running the script and then a few minutes doing CSG in CGAL.

I also recently moved all my test code to their own file, so my used files don't have any tests in them. I did this so that l can automatically run them to produce images.

The reason include would cause my files to be parsed multiple times is because a lot of them are used in every other library file. This would lead to them being parsed tens of thousands of times due to tree explosion. I know this because I had to fix the module cache dependency checking which was calling stat ten thousand times.

So I have a very different use case to Adrian. Most of my code is making models of real world objects with CSG. Only a few need swept polyhedra and even that only calls functions hundreds of times, not thousands.

Other functions like screw_clearance_radius get called for each screw in a project, and there are only hundreds of those in a large project.

On Wed, 17 Apr 2019, 01:32 A. Craig West, <[hidden email]> wrote:
I believe "include" doesn't properly track relative directories when including other files, while "use" does, although this may have been resolved at some point. I haven't checked lately

On Tue, 16 Apr 2019, 20:12 Hugo Jackson, <[hidden email]> wrote:
> It does appear that the obvious solution is to simply ignore "use" and
> always use "include".  Is there some disadvantage to this approach?  I

I use “use” over include in that it allows me to keep test invocations in library files. Include as you know will execute any 0-level commands it encounters when being included… where as “use” only loads the function/module definitions and executes nothing.


> suppose you lose the ability to "use" files from a library and not expose
> their contents to the to level "user".  Nophead reported a slight
> slowdown...do to reparsing?  I'm not sure I understand why that would
> happen. 
>
>
>
>
> --
> Sent from: http://forum.openscad.org/
>
> _______________________________________________
> OpenSCAD mailing list
> [hidden email]
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

acwest
The other use case for use vs include is dependencies. There doesn't appear to be a way to prevent multiple inclusions of the same file. I'm not sure if it helps, but at least in theory use is better in this case. I don't know how to prevent circular dependencies, though. 

On Wed, 17 Apr 2019, 03:58 nop head, <[hidden email]> wrote:
The reason l haven't noticed is that l simply don't have thousands of function calls across file boundaries. My projects typically spend a second or two running the script and then a few minutes doing CSG in CGAL.

I also recently moved all my test code to their own file, so my used files don't have any tests in them. I did this so that l can automatically run them to produce images.

The reason include would cause my files to be parsed multiple times is because a lot of them are used in every other library file. This would lead to them being parsed tens of thousands of times due to tree explosion. I know this because I had to fix the module cache dependency checking which was calling stat ten thousand times.

So I have a very different use case to Adrian. Most of my code is making models of real world objects with CSG. Only a few need swept polyhedra and even that only calls functions hundreds of times, not thousands.

Other functions like screw_clearance_radius get called for each screw in a project, and there are only hundreds of those in a large project.

On Wed, 17 Apr 2019, 01:32 A. Craig West, <[hidden email]> wrote:
I believe "include" doesn't properly track relative directories when including other files, while "use" does, although this may have been resolved at some point. I haven't checked lately

On Tue, 16 Apr 2019, 20:12 Hugo Jackson, <[hidden email]> wrote:
> It does appear that the obvious solution is to simply ignore "use" and
> always use "include".  Is there some disadvantage to this approach?  I

I use “use” over include in that it allows me to keep test invocations in library files. Include as you know will execute any 0-level commands it encounters when being included… where as “use” only loads the function/module definitions and executes nothing.


> suppose you lose the ability to "use" files from a library and not expose
> their contents to the to level "user".  Nophead reported a slight
> slowdown...do to reparsing?  I'm not sure I understand why that would
> happen. 
>
>
>
>
> --
> Sent from: http://forum.openscad.org/
>
> _______________________________________________
> OpenSCAD mailing list
> [hidden email]
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

nophead
Yes used files are cached so a second use simple looks up the name and finds it already parsed in the cache.

On Thu, 18 Apr 2019, 09:37 A. Craig West, <[hidden email]> wrote:
The other use case for use vs include is dependencies. There doesn't appear to be a way to prevent multiple inclusions of the same file. I'm not sure if it helps, but at least in theory use is better in this case. I don't know how to prevent circular dependencies, though. 

On Wed, 17 Apr 2019, 03:58 nop head, <[hidden email]> wrote:
The reason l haven't noticed is that l simply don't have thousands of function calls across file boundaries. My projects typically spend a second or two running the script and then a few minutes doing CSG in CGAL.

I also recently moved all my test code to their own file, so my used files don't have any tests in them. I did this so that l can automatically run them to produce images.

The reason include would cause my files to be parsed multiple times is because a lot of them are used in every other library file. This would lead to them being parsed tens of thousands of times due to tree explosion. I know this because I had to fix the module cache dependency checking which was calling stat ten thousand times.

So I have a very different use case to Adrian. Most of my code is making models of real world objects with CSG. Only a few need swept polyhedra and even that only calls functions hundreds of times, not thousands.

Other functions like screw_clearance_radius get called for each screw in a project, and there are only hundreds of those in a large project.

On Wed, 17 Apr 2019, 01:32 A. Craig West, <[hidden email]> wrote:
I believe "include" doesn't properly track relative directories when including other files, while "use" does, although this may have been resolved at some point. I haven't checked lately

On Tue, 16 Apr 2019, 20:12 Hugo Jackson, <[hidden email]> wrote:
> It does appear that the obvious solution is to simply ignore "use" and
> always use "include".  Is there some disadvantage to this approach?  I

I use “use” over include in that it allows me to keep test invocations in library files. Include as you know will execute any 0-level commands it encounters when being included… where as “use” only loads the function/module definitions and executes nothing.


> suppose you lose the ability to "use" files from a library and not expose
> their contents to the to level "user".  Nophead reported a slight
> slowdown...do to reparsing?  I'm not sure I understand why that would
> happen. 
>
>
>
>
> --
> Sent from: http://forum.openscad.org/
>
> _______________________________________________
> OpenSCAD mailing list
> [hidden email]
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

adrianv
In reply to this post by boxcarmib
boxcarmib wrote
>> It does appear that the obvious solution is to simply ignore "use" and
>> always use "include".  Is there some disadvantage to this approach?  I
>
> I use “use” over include in that it allows me to keep test invocations in
> library files. Include as you know will execute any 0-level commands it
> encounters when being included… where as “use” only loads the
> function/module definitions and executes nothing.

Yeah, that's the theory.  But since it doesn't actually work that way it not
such a clear advantage.   I guess it depends on whether your test
invocations assign any variables.   In the file I was working using, the
presence test invocations (which made top level variable assignments) slowed
down my code from 5 s to 10.5 minutes.  




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

adrianv
In reply to this post by nophead
There are several things I don't understand here.  if my to level does
"use<library.scad>" and that file in turn has the statement
"use<sublib.scad>" then you avoid namespace clutter because sublib.scad is
not made available to the top level.  
What happens if I have lib1.scad, lib2.scad, ..., .libN.scad and they all
run "use<sublib.scad>"?  Does sublib.scad get read in N times?  Or it's in
the cache and it doesn't need to be re-read?  

I would assume that if you changed "use" to "include" in this example then
definitely the file would be read every time.  

I don't understand the issue about dependencies.   What is the concern?
With "include" the file gets read again and the definitions replace the old
values (which are the same), right?   I tried creating circular "use"
statements and everything seemed to work.  With circular "include"
statements I get "Can't open include file...."  But I can't tell if it
loaded the file a hundred times or what.  Is there a max depth?  Are
circular library dependencies a real concern?  

If "used" files are cached then why is it so much slower to call functions
in a large used library?  I am finding the behavior pretty odd.  I tried
tweaking my example as follows.  I placed my quant() function in a separate
file, so I have basically

use<quant.scad>
use<hugelib.scad>

This runs fast.  So just creating a large cache with a big library doesn't
create a slow down.   If I define quant() in *both* files then it depends on
the "use" order.   If hugelib is second then it is slow.  If quant.scad is
second, it's fast.  The last definition holds.  I would expect that if
definitions are cached then a slowdown would have to depend on cache size,
and that wouldn't depend on the order of "use" statements.  Is there a
separate cache entry for each "used" file and a large entry (lots of
functions) makes retrieval slow?   How is it different when a function is in
this cache vs just being defined at the top level?  Where does it go then?  

To get ten thousand references to the same file due to the include tree
would seem to require a very deep include tree with lots of files all using
files that use other files that use the same thing.   Is that how your code
is structured?   The code I'm working on the library files are including
files to only one more level of depth.  


nophead wrote
> Yes used files are cached so a second use simple looks up the name and
> finds it already parsed in the cache.
>
> On Thu, 18 Apr 2019, 09:37 A. Craig West, &lt;

> acraigwest@

> &gt; wrote:
>
>> The other use case for use vs include is dependencies. There doesn't
>> appear to be a way to prevent multiple inclusions of the same file. I'm
>> not
>> sure if it helps, but at least in theory use is better in this case. I
>> don't know how to prevent circular dependencies, though.
>>
>> On Wed, 17 Apr 2019, 03:58 nop head, &lt;

> nop.head@

> &gt; wrote:
>>
>>> The reason l haven't noticed is that l simply don't have thousands of
>>> function calls across file boundaries. My projects typically spend a
>>> second
>>> or two running the script and then a few minutes doing CSG in CGAL.
>>>
>>> I also recently moved all my test code to their own file, so my used
>>> files don't have any tests in them. I did this so that l can
>>> automatically
>>> run them to produce images.
>>>
>>> The reason include would cause my files to be parsed multiple times is
>>> because a lot of them are used in every other library file. This would
>>> lead
>>> to them being parsed tens of thousands of times due to tree explosion. I
>>> know this because I had to fix the module cache dependency checking
>>> which
>>> was calling stat ten thousand times.
>>>
>>> So I have a very different use case to Adrian. Most of my code is making
>>> models of real world objects with CSG. Only a few need swept polyhedra
>>> and
>>> even that only calls functions hundreds of times, not thousands.
>>>
>>> Other functions like screw_clearance_radius get called for each screw in
>>> a project, and there are only hundreds of those in a large project.
>>>
>>> On Wed, 17 Apr 2019, 01:32 A. Craig West, &lt;

> acraigwest@

> &gt; wrote:
>>>
>>>> I believe "include" doesn't properly track relative directories when
>>>> including other files, while "use" does, although this may have been
>>>> resolved at some point. I haven't checked lately
>>>>
>>>> On Tue, 16 Apr 2019, 20:12 Hugo Jackson, &lt;

> hugo@

> &gt; wrote:
>>>>
>>>>> > It does appear that the obvious solution is to simply ignore "use"
>>>>> and
>>>>> > always use "include".  Is there some disadvantage to this approach?
>>>>> I
>>>>>
>>>>> I use “use” over include in that it allows me to keep test invocations
>>>>> in library files. Include as you know will execute any 0-level
>>>>> commands it
>>>>> encounters when being included… where as “use” only loads the
>>>>> function/module definitions and executes nothing.
>>>>>
>>>>>
>>>>> > suppose you lose the ability to "use" files from a library and not
>>>>> expose
>>>>> > their contents to the to level "user".  Nophead reported a slight
>>>>> > slowdown...do to reparsing?  I'm not sure I understand why that
>>>>> would
>>>>> > happen.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Sent from: http://forum.openscad.org/
>>>>> >
>>>>> > _______________________________________________
>>>>> > OpenSCAD mailing list
>>>>> >

> Discuss@.openscad

>>>>> >
>>>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenSCAD mailing list
>>>>>

> Discuss@.openscad

>>>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>>>
>>>> _______________________________________________
>>>> OpenSCAD mailing list
>>>>

> Discuss@.openscad

>>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>>
>>> _______________________________________________
>>> OpenSCAD mailing list
>>>

> Discuss@.openscad

>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>
>> _______________________________________________
>> OpenSCAD mailing list
>>

> Discuss@.openscad

>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>
>
> _______________________________________________
> OpenSCAD mailing list

> Discuss@.openscad

> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org





--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: "use" results in 3x more run time than "include"

nophead
With use there is only one module cache entry for each used file and it is only read from file once and parsed once regardless of how may times it is used and from where.

They way I get a tree explosion is having a project where each file uses all the files in my library. All all the files in my library use a global file that uses several itself and several other library files. So it creates a sort of fractal tree.

Symbol lookup is pretty quick I think because it uses maps. But if you have a list of modules then it has to look for symbols in each one. So lots of maps searches.

If you include then it gets parsed over and over again and I think creates lots of copies of the functions and modules in different namespaces.

If you have recursive uses it simply ignores the recursion because it finds in the cache. On the other hand with a recursive include it would loop forever if it didn't error.

On Thu, 18 Apr 2019 at 13:01, adrianv <[hidden email]> wrote:
There are several things I don't understand here.  if my to level does
"use<library.scad>" and that file in turn has the statement
"use<sublib.scad>" then you avoid namespace clutter because sublib.scad is
not made available to the top level. 
What happens if I have lib1.scad, lib2.scad, ..., .libN.scad and they all
run "use<sublib.scad>"?  Does sublib.scad get read in N times?  Or it's in
the cache and it doesn't need to be re-read? 

I would assume that if you changed "use" to "include" in this example then
definitely the file would be read every time. 

I don't understand the issue about dependencies.   What is the concern?
With "include" the file gets read again and the definitions replace the old
values (which are the same), right?   I tried creating circular "use"
statements and everything seemed to work.  With circular "include"
statements I get "Can't open include file...."  But I can't tell if it
loaded the file a hundred times or what.  Is there a max depth?  Are
circular library dependencies a real concern? 

If "used" files are cached then why is it so much slower to call functions
in a large used library?  I am finding the behavior pretty odd.  I tried
tweaking my example as follows.  I placed my quant() function in a separate
file, so I have basically

use<quant.scad>
use<hugelib.scad>

This runs fast.  So just creating a large cache with a big library doesn't
create a slow down.   If I define quant() in *both* files then it depends on
the "use" order.   If hugelib is second then it is slow.  If quant.scad is
second, it's fast.  The last definition holds.  I would expect that if
definitions are cached then a slowdown would have to depend on cache size,
and that wouldn't depend on the order of "use" statements.  Is there a
separate cache entry for each "used" file and a large entry (lots of
functions) makes retrieval slow?   How is it different when a function is in
this cache vs just being defined at the top level?  Where does it go then? 

To get ten thousand references to the same file due to the include tree
would seem to require a very deep include tree with lots of files all using
files that use other files that use the same thing.   Is that how your code
is structured?   The code I'm working on the library files are including
files to only one more level of depth. 


nophead wrote
> Yes used files are cached so a second use simple looks up the name and
> finds it already parsed in the cache.
>
> On Thu, 18 Apr 2019, 09:37 A. Craig West, &lt;

> acraigwest@

> &gt; wrote:
>
>> The other use case for use vs include is dependencies. There doesn't
>> appear to be a way to prevent multiple inclusions of the same file. I'm
>> not
>> sure if it helps, but at least in theory use is better in this case. I
>> don't know how to prevent circular dependencies, though.
>>
>> On Wed, 17 Apr 2019, 03:58 nop head, &lt;

> nop.head@

> &gt; wrote:
>>
>>> The reason l haven't noticed is that l simply don't have thousands of
>>> function calls across file boundaries. My projects typically spend a
>>> second
>>> or two running the script and then a few minutes doing CSG in CGAL.
>>>
>>> I also recently moved all my test code to their own file, so my used
>>> files don't have any tests in them. I did this so that l can
>>> automatically
>>> run them to produce images.
>>>
>>> The reason include would cause my files to be parsed multiple times is
>>> because a lot of them are used in every other library file. This would
>>> lead
>>> to them being parsed tens of thousands of times due to tree explosion. I
>>> know this because I had to fix the module cache dependency checking
>>> which
>>> was calling stat ten thousand times.
>>>
>>> So I have a very different use case to Adrian. Most of my code is making
>>> models of real world objects with CSG. Only a few need swept polyhedra
>>> and
>>> even that only calls functions hundreds of times, not thousands.
>>>
>>> Other functions like screw_clearance_radius get called for each screw in
>>> a project, and there are only hundreds of those in a large project.
>>>
>>> On Wed, 17 Apr 2019, 01:32 A. Craig West, &lt;

> acraigwest@

> &gt; wrote:
>>>
>>>> I believe "include" doesn't properly track relative directories when
>>>> including other files, while "use" does, although this may have been
>>>> resolved at some point. I haven't checked lately
>>>>
>>>> On Tue, 16 Apr 2019, 20:12 Hugo Jackson, &lt;

> hugo@

> &gt; wrote:
>>>>
>>>>> > It does appear that the obvious solution is to simply ignore "use"
>>>>> and
>>>>> > always use "include".  Is there some disadvantage to this approach?
>>>>> I
>>>>>
>>>>> I use “use” over include in that it allows me to keep test invocations
>>>>> in library files. Include as you know will execute any 0-level
>>>>> commands it
>>>>> encounters when being included… where as “use” only loads the
>>>>> function/module definitions and executes nothing.
>>>>>
>>>>>
>>>>> > suppose you lose the ability to "use" files from a library and not
>>>>> expose
>>>>> > their contents to the to level "user".  Nophead reported a slight
>>>>> > slowdown...do to reparsing?  I'm not sure I understand why that
>>>>> would
>>>>> > happen.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Sent from: http://forum.openscad.org/
>>>>> >
>>>>> > _______________________________________________
>>>>> > OpenSCAD mailing list
>>>>> >

> Discuss@.openscad

>>>>> >
>>>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenSCAD mailing list
>>>>>

> Discuss@.openscad

>>>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>>>
>>>> _______________________________________________
>>>> OpenSCAD mailing list
>>>>

> Discuss@.openscad

>>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>>
>>> _______________________________________________
>>> OpenSCAD mailing list
>>>

> Discuss@.openscad

>>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>>
>> _______________________________________________
>> OpenSCAD mailing list
>>

> Discuss@.openscad

>> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>>
>
> _______________________________________________
> OpenSCAD mailing list

> Discuss@.openscad

> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org





--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
12