Problem with large projects

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with large projects

nophead
I have been designing a 3D printer on and off for a few years now. When I came back to it recently, haing updated my library and OpenSCAD in the meantime, I find it very slow to update the preview when I make a small change. This is mainly the "Compiling design (CSG Tree generation)...", i.e. when the code is instantiated and echo modules come out, before it actually does any geometry. The geometry takes no time when nothing has changed because it is all cached.

I remember it being about 12 seconds, which wasn't too bad but now it is a minute. 
 
It is by far the biggest project I have done in OpenSCAD, and is broken up into 21 files, not including the library. In contrast most of other projects are just a single file or two or three and will do that part of the compilation in less than a second. For example something like this uses the same library and will do F5 with no changes in 1 second.

image.png

It only takes 53 seconds to draw the whole thing when first loaded. My large project only takes 4 minutes to render from scratch, so whereas the small project has a 53:1 difference between the first F5 and a second one. The large project only has a 4:1 difference.

It is hard to see where the time is taken, it seems to be fairly slow all over just due to splitting it into multiple files. I think the main problem is a small file of global constants that gets included everywhere. Putting an echo function in that file shows the constants are instantiated about 1/4 million times! In the smaller project it is instantiated 4700 times. Still huge but it seems to explode when you have more files.

This is because whenever you call a module or function in a different file it instantiates all the constants. Is this really necessary or can it be fixed? I think it would make a huge difference to a project like this.







_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

MostlyHarmless


On Sun, Jun 7, 2020 at 12:55 PM nop head <[hidden email]> wrote:
I have been designing a 3D printer on and off for a few years now. When I came back to it recently, haing updated my library and OpenSCAD in the meantime, I find it very slow to update the preview when I make a small change. This is mainly the "Compiling design (CSG Tree generation)...", i.e. when the code is instantiated and echo modules come out, before it actually does any geometry. The geometry takes no time when nothing has changed because it is all cached.

Considering that this is all CPU bound, have you tried getting a perf based CPU flamegraph of the rendering process? My current OpenSCAD setup is living in a docker container and I have never attempted to perf such, but it would be an interesting thing to figure out.


Regards, Jan



 

I remember it being about 12 seconds, which wasn't too bad but now it is a minute. 
 
It is by far the biggest project I have done in OpenSCAD, and is broken up into 21 files, not including the library. In contrast most of other projects are just a single file or two or three and will do that part of the compilation in less than a second. For example something like this uses the same library and will do F5 with no changes in 1 second.

image.png

It only takes 53 seconds to draw the whole thing when first loaded. My large project only takes 4 minutes to render from scratch, so whereas the small project has a 53:1 difference between the first F5 and a second one. The large project only has a 4:1 difference.

It is hard to see where the time is taken, it seems to be fairly slow all over just due to splitting it into multiple files. I think the main problem is a small file of global constants that gets included everywhere. Putting an echo function in that file shows the constants are instantiated about 1/4 million times! In the smaller project it is instantiated 4700 times. Still huge but it seems to explode when you have more files.

This is because whenever you call a module or function in a different file it instantiates all the constants. Is this really necessary or can it be fixed? I think it would make a huge difference to a project like this.






_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


--
Jan Wieck
Principal Database Engineer

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

adrianv
In reply to this post by nophead
Yeah, when I posted about this a few months ago people seemed to think it
didn't matter.  The issue has been open for years and seems catastrophic to
me.  

Our solution is to never use "use" and only use "include" which avoids the
problem.  I think there are other cases where "use" is slow as well, though
not as serious.  




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

nophead
Yes I remember that post. I seem to get away with it on all my projects except this one. It also has a global file with constants at the project level and they are positions of things with several function calls each to other modules, so they are more costly. Each of the function calls likely needs all the constants instantiated again and it explodes.

I can get the time down to 37 seconds by including sub assemblies instead of using them but I had to add a bodge to stop them instantiating themselves as I like to load a sub assembly when working on it.

I am on Windows so to profile OpenSCAD I think I would need to build it with Visual Studio, which I have done before, but I am pretty sure the problem is evaluating constants over and over again. Every single screw, nut, washer, etc are calls to the library and the library includes its global constants in every module as again, they all work standalone.  

It is counter intuitive that making constants in a file module is slower than working everything out inside functions. You would expect the constant to be evaluated once when the file is first used, not every time a function or module is called in it.

It might be faster to make all constants into functions but then you would have a big tree of function calls that explode as nothing can be worked out once and stored for later use.


On Sun, 7 Jun 2020 at 18:33, adrianv <[hidden email]> wrote:
Yeah, when I posted about this a few months ago people seemed to think it
didn't matter.  The issue has been open for years and seems catastrophic to
me. 

Our solution is to never use "use" and only use "include" which avoids the
problem.  I think there are other cases where "use" is slow as well, though
not as serious. 




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

nophead
Yes this has really become a problem for me. I have a list of transformations that position screws in a shelf bracket that is reasonably complex to calculate and since it generates the bracket and also positions inserts and screws into it plus it needs to drill holes in 6 panels that it mates with I need to reference it many times.

function shelf_screw_positions() = [
    translate([z_axis_x + spine_width() / 2, box_depth / 2 - spine_depth() + spine_clearance + shelf_boss_r + eps])               * rotate([-90, 0, 90]),
    translate([z_axis_x + spine_width() / 2, box_depth / 2 - shelf_boss_r - eps])                                                 * rotate([-90, 0, 90]),

    translate([-box_width / 4,               box_depth / 2])                                                                      * rotate([-90, 0,  0]),

    translate([box_width / 2,                box_depth / 2 - box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) - eps]) * rotate([-90, 0, -90]),
    translate([box_width / 2,               -box_depth / 2 + box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) + eps]) * rotate([-90, 0, -90]),

    translate([ 10, -box_depth / 2]) * rotate([-90, 0, 180]),
    translate([-10, -box_depth / 2]) * rotate([-90, 0, 180]),

    translate([-box_width / 2,               box_depth / 2 - box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) - eps]) * rotate([-90, 0, 90]),
    translate([-box_width / 2,              -box_depth / 2 + box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) + eps]) * rotate([-90, 0, 90]),

    translate([z_axis_x - spine_width() / 2, box_depth / 2 - spine_depth() + spine_clearance + shelf_boss_r + eps])               * rotate([-90, 0, -90]),
    translate([z_axis_x - spine_width() / 2, box_depth / 2 - shelf_boss_r - eps])                                                 * rotate([-90, 0, -90]),
];


image.png

If I store it in a variable it gets calculated every time I call a function or module in a file that includes its definition. If I wrap it in a function it gets computed every time I use it. It is bonkers because it is a constant, so only needs to be calculated once.

Adding this shelf to my project doubles the time it takes to do "Compiling design (CSG Tree generation)...", from about 30 seconds to 1 minute simply because this self touches six panels, which are all separate files in the machine. Compared to the rest of the complexity of the machine it is trivial but because it interacts with most of it it has a huge impact on the speed and the time needed to see every change.

Is it true that the only way to have a constant calculated once is to make my project into one massive file?


On Sun, 7 Jun 2020 at 21:03, nop head <[hidden email]> wrote:
Yes I remember that post. I seem to get away with it on all my projects except this one. It also has a global file with constants at the project level and they are positions of things with several function calls each to other modules, so they are more costly. Each of the function calls likely needs all the constants instantiated again and it explodes.

I can get the time down to 37 seconds by including sub assemblies instead of using them but I had to add a bodge to stop them instantiating themselves as I like to load a sub assembly when working on it.

I am on Windows so to profile OpenSCAD I think I would need to build it with Visual Studio, which I have done before, but I am pretty sure the problem is evaluating constants over and over again. Every single screw, nut, washer, etc are calls to the library and the library includes its global constants in every module as again, they all work standalone.  

It is counter intuitive that making constants in a file module is slower than working everything out inside functions. You would expect the constant to be evaluated once when the file is first used, not every time a function or module is called in it.

It might be faster to make all constants into functions but then you would have a big tree of function calls that explode as nothing can be worked out once and stored for later use.


On Sun, 7 Jun 2020 at 18:33, adrianv <[hidden email]> wrote:
Yeah, when I posted about this a few months ago people seemed to think it
didn't matter.  The issue has been open for years and seems catastrophic to
me. 

Our solution is to never use "use" and only use "include" which avoids the
problem.  I think there are other cases where "use" is slow as well, though
not as serious. 




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

Peter
I have my own "constructive" library (kind of a strong dialect of openscad entirely written in openscad)  i have been using for many years now for complex projects.
in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?
will this help the case? i am not 100% sure i understand your case completely.
i planned to publish it anyway, but since the reaction to my work in this group so far was never very enthusiastic, it seems it will probably not pay of the effort of cleaning up and documenting the code.
but in case it  might help you with your problem i am happy to share it with you, it has many other useful things like automatic backwards(reverse) transformations, ability to mark parts of the code as relevant to one design part or other, and then, render only certain parts, or boolean them, or having "difference()" work over different transformation origins (the components of the difference operation do not have to have the same parent,like with normal difference() where the subtracted componets are always ancestors of the difference() element),selective hull(),...
 
On 15.06.20 01:11, nop head wrote:
Yes this has really become a problem for me. I have a list of transformations that position screws in a shelf bracket that is reasonably complex to calculate and since it generates the bracket and also positions inserts and screws into it plus it needs to drill holes in 6 panels that it mates with I need to reference it many times.

function shelf_screw_positions() = [
    translate([z_axis_x + spine_width() / 2, box_depth / 2 - spine_depth() + spine_clearance + shelf_boss_r + eps])               * rotate([-90, 0, 90]),
    translate([z_axis_x + spine_width() / 2, box_depth / 2 - shelf_boss_r - eps])                                                 * rotate([-90, 0, 90]),

    translate([-box_width / 4,               box_depth / 2])                                                                      * rotate([-90, 0,  0]),

    translate([box_width / 2,                box_depth / 2 - box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) - eps]) * rotate([-90, 0, -90]),
    translate([box_width / 2,               -box_depth / 2 + box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) + eps]) * rotate([-90, 0, -90]),

    translate([ 10, -box_depth / 2]) * rotate([-90, 0, 180]),
    translate([-10, -box_depth / 2]) * rotate([-90, 0, 180]),

    translate([-box_width / 2,               box_depth / 2 - box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) - eps]) * rotate([-90, 0, 90]),
    translate([-box_width / 2,              -box_depth / 2 + box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) + eps]) * rotate([-90, 0, 90]),

    translate([z_axis_x - spine_width() / 2, box_depth / 2 - spine_depth() + spine_clearance + shelf_boss_r + eps])               * rotate([-90, 0, -90]),
    translate([z_axis_x - spine_width() / 2, box_depth / 2 - shelf_boss_r - eps])                                                 * rotate([-90, 0, -90]),
];


image.png

If I store it in a variable it gets calculated every time I call a function or module in a file that includes its definition. If I wrap it in a function it gets computed every time I use it. It is bonkers because it is a constant, so only needs to be calculated once.

Adding this shelf to my project doubles the time it takes to do "Compiling design (CSG Tree generation)...", from about 30 seconds to 1 minute simply because this self touches six panels, which are all separate files in the machine. Compared to the rest of the complexity of the machine it is trivial but because it interacts with most of it it has a huge impact on the speed and the time needed to see every change.

Is it true that the only way to have a constant calculated once is to make my project into one massive file?


On Sun, 7 Jun 2020 at 21:03, nop head <[hidden email]> wrote:
Yes I remember that post. I seem to get away with it on all my projects except this one. It also has a global file with constants at the project level and they are positions of things with several function calls each to other modules, so they are more costly. Each of the function calls likely needs all the constants instantiated again and it explodes.

I can get the time down to 37 seconds by including sub assemblies instead of using them but I had to add a bodge to stop them instantiating themselves as I like to load a sub assembly when working on it.

I am on Windows so to profile OpenSCAD I think I would need to build it with Visual Studio, which I have done before, but I am pretty sure the problem is evaluating constants over and over again. Every single screw, nut, washer, etc are calls to the library and the library includes its global constants in every module as again, they all work standalone.  

It is counter intuitive that making constants in a file module is slower than working everything out inside functions. You would expect the constant to be evaluated once when the file is first used, not every time a function or module is called in it.

It might be faster to make all constants into functions but then you would have a big tree of function calls that explode as nothing can be worked out once and stored for later use.


On Sun, 7 Jun 2020 at 18:33, adrianv <[hidden email]> wrote:
Yeah, when I posted about this a few months ago people seemed to think it
didn't matter.  The issue has been open for years and seems catastrophic to
me. 

Our solution is to never use "use" and only use "include" which avoids the
problem.  I think there are other cases where "use" is slow as well, though
not as serious. 




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

Ronaldo
in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?

As far as I know, they will not be cached. Only solids, models, objects are cached. Variables are not.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

nophead
Yes that is the problem. If you have some complex code that generates a solid it is all run on every update, the parameters it generates are then compared with the cached solids to see if the geometry needs updating. In my project that takes no time at all but running all the code to generate thousands of parts starts to take a long time. I think this is mainly because every constant that isn't in your top level file gets evaluated over and over again. 

For isolation I have individual files that I can load on their own, for example the z axis assembly, which is quite complex. At the start of the file it has a lot of constants to work out positions, etc. It also includes definitions of all the vitamins it uses, which are hundreds of lists.  It also needs to export a few functions and modules to the rest of the machine, which just export constants, like the total height. However calling any of these means all the hundreds of constants it uses are all evaluated over and over again. So even on a PC it starts to take significant time. It really shouldn't take that long to evaluate a script but the constant evaluation explosion does as soon as you separate a project into lots of files.

On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano <[hidden email]> wrote:
in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?

As far as I know, they will not be cached. Only solids, models, objects are cached. Variables are not.
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

Peter
i think one possible approach would be to do big Part of the  caching yourself, that is use lookup tables.
so if all transformation sequences are accumulated into single matrices,
i would create a vector of matrices of all transformations you will need, and write it out as long string into lookup.scad file
 this lookup.scad containing only single assignment of the lookup vector of matrices: lookup=[[1,2,2,2,...}[5,34,887,87,..], yadda yadda];
then just include the lookup.scad and use the multmatrix(lookup[indexofNeededVal])
in the main code
creating lookup tables is never the most elegant approach,but still, flexible enough, and will probably save a lot of rendering time. 

On 15.06.20 10:17, nop head wrote:
Yes that is the problem. If you have some complex code that generates a solid it is all run on every update, the parameters it generates are then compared with the cached solids to see if the geometry needs updating. In my project that takes no time at all but running all the code to generate thousands of parts starts to take a long time. I think this is mainly because every constant that isn't in your top level file gets evaluated over and over again. 

For isolation I have individual files that I can load on their own, for example the z axis assembly, which is quite complex. At the start of the file it has a lot of constants to work out positions, etc. It also includes definitions of all the vitamins it uses, which are hundreds of lists.  It also needs to export a few functions and modules to the rest of the machine, which just export constants, like the total height. However calling any of these means all the hundreds of constants it uses are all evaluated over and over again. So even on a PC it starts to take significant time. It really shouldn't take that long to evaluate a script but the constant evaluation explosion does as soon as you separate a project into lots of files.

On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano <[hidden email]> wrote:
in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?

As far as I know, they will not be cached. Only solids, models, objects are cached. Variables are not.
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

nophead
But it is then no longer parametric, the main reason for using OpenSCAD.

It isn't particularly slow to multiply matrices. It is only when it is done over and over again. I sped it up considerably by caching functions calls like this.

function shelf_screw_positions() = let(sw = spine_width(), sd = spine_depth(), bc = box_intrusion(box_type) + box_bezel_clearance(box_type)) [
    translate([z_axis_x + sw / 2, box_depth / 2 - sd + spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0, 90]),
    translate([z_axis_x + sw / 2, box_depth / 2 - shelf_boss_r - eps])                        * rotate([-90, 0, 90]),

    translate([-box_width / 4,               box_depth / 2])                                  * rotate([-90, 0,  0]),

    translate([box_width / 2,                box_depth / 2 - bc - shelf_boss_r - eps])        * rotate([-90, 0, -90]),
    translate([box_width / 2,               -box_depth / 2 + bc + shelf_boss_r + eps])        * rotate([-90, 0, -90]),

    translate([ 10, -box_depth / 2])                                                          * rotate([-90, 0, 180]),
    translate([-10, -box_depth / 2])                                                          * rotate([-90, 0, 180]),

    translate([-box_width / 2,               box_depth / 2 - bc - shelf_boss_r - eps])        * rotate([-90, 0, 90]),
    translate([-box_width / 2,              -box_depth / 2 + bc + shelf_boss_r + eps])        * rotate([-90, 0, 90]),

    translate([z_axis_x - sw / 2, box_depth / 2 - sd + spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0, -90]),
    translate([z_axis_x - sw / 2, box_depth / 2 - shelf_boss_r - eps])                        * rotate([-90, 0, -90]),
];


The three functions I calculate once in the let() and then reuse two or three times are trivial functions but they are slow because they are in a used file that has lots of constants defined in it. So each time they are called lots of constants are evaluated again. Also this function should be a constant, not a function, but then it gets evaluated whenever something else is used from the file that defines it.

I got a  big speed up making it a function until I added more code that called it, so I can't win. There is no way to calculate a constant once and store it except to do it in the main file, but then it can't be accessed from any other file unless they are all included. Then they can't be used standalone and that breaks my build system because it locates stls and assemblies in the file that defines them and calls them from the command line. So each file has to be self contained and include or use all its dependencies.

On Mon, 15 Jun 2020 at 12:37, [hidden email] <[hidden email]> wrote:
i think one possible approach would be to do big Part of the  caching yourself, that is use lookup tables.
so if all transformation sequences are accumulated into single matrices,
i would create a vector of matrices of all transformations you will need, and write it out as long string into lookup.scad file
 this lookup.scad containing only single assignment of the lookup vector of matrices: lookup=[[1,2,2,2,...}[5,34,887,87,..], yadda yadda];
then just include the lookup.scad and use the multmatrix(lookup[indexofNeededVal])
in the main code
creating lookup tables is never the most elegant approach,but still, flexible enough, and will probably save a lot of rendering time. 

On 15.06.20 10:17, nop head wrote:
Yes that is the problem. If you have some complex code that generates a solid it is all run on every update, the parameters it generates are then compared with the cached solids to see if the geometry needs updating. In my project that takes no time at all but running all the code to generate thousands of parts starts to take a long time. I think this is mainly because every constant that isn't in your top level file gets evaluated over and over again. 

For isolation I have individual files that I can load on their own, for example the z axis assembly, which is quite complex. At the start of the file it has a lot of constants to work out positions, etc. It also includes definitions of all the vitamins it uses, which are hundreds of lists.  It also needs to export a few functions and modules to the rest of the machine, which just export constants, like the total height. However calling any of these means all the hundreds of constants it uses are all evaluated over and over again. So even on a PC it starts to take significant time. It really shouldn't take that long to evaluate a script but the constant evaluation explosion does as soon as you separate a project into lots of files.

On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano <[hidden email]> wrote:
in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?

As far as I know, they will not be cached. Only solids, models, objects are cached. Variables are not.
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

jon_bondy
In reply to this post by nophead

Would there be any advantage to temporarily linearizing (combining) the files just for the purpose of compilation?  Many people run external editors and only use OpenSCAD as a compiler.  Suppose the editor automatically created a file called Linearized.SCAD and that was what OpenSCAD focused on, leaving the structure of your source code intact for editing.  Quite a hack, but would there be any benefits?

On 6/15/2020 4:17 AM, nop head wrote:
Yes that is the problem. If you have some complex code that generates a solid it is all run on every update, the parameters it generates are then compared with the cached solids to see if the geometry needs updating. In my project that takes no time at all but running all the code to generate thousands of parts starts to take a long time. I think this is mainly because every constant that isn't in your top level file gets evaluated over and over again. 

For isolation I have individual files that I can load on their own, for example the z axis assembly, which is quite complex. At the start of the file it has a lot of constants to work out positions, etc. It also includes definitions of all the vitamins it uses, which are hundreds of lists.  It also needs to export a few functions and modules to the rest of the machine, which just export constants, like the total height. However calling any of these means all the hundreds of constants it uses are all evaluated over and over again. So even on a PC it starts to take significant time. It really shouldn't take that long to evaluate a script but the constant evaluation explosion does as soon as you separate a project into lots of files.

On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano <[hidden email]> wrote:
in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?

As far as I know, they will not be cached. Only solids, models, objects are cached. Variables are not.
_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

jon_bondy

Were you to manually put all of the files into a single temporary file, and compile that temporary file, what speed benefit would you see?  If the benefit is large, why not write a program to do all of the <include>s and <uses> automatically and then only use separate files for writing and the automatically generated temporary file  for compiling? 

On 6/15/2020 8:16 AM, nop head wrote:
Well I included a lot of the files that were used by main but then they all include the global definitions instead of just using the ones at the top of main. That did give a speed up but is not optimum without removing all the extra includes.

Some time ago the main file had most of the machine in it and was a lot faster but was huge and hard to navigate. With other languages you don't get a huge penalty when you break the problem into smaller modules. OpenSCAD has a massive bug where calling a function or a module in a used file recalculates all the constants. I think it only needs to calculate any that involve $variables, (directly or indirectly) that have changed since the last time they were evaluated and that would affect the function or module being called. So not an easy bug to fix.
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

nophead
When I made the change it went from a minute to about 30 seconds. When I added all the code to drill the screw holes in all the panels that mate with the shelf bracket it slowed down to a minute again. Caching the three function calls got it down to 45 seconds.

The shelf bracket is pathological because it depends on all the sides and affects all the sides. It is also too big to print so needs to be slit into sections. That will add more screw holes so it will end up even slower.

On Mon, 15 Jun 2020 at 13:27, jon <[hidden email]> wrote:

Were you to manually put all of the files into a single temporary file, and compile that temporary file, what speed benefit would you see?  If the benefit is large, why not write a program to do all of the <include>s and <uses> automatically and then only use separate files for writing and the automatically generated temporary file  for compiling? 

On 6/15/2020 8:16 AM, nop head wrote:
Well I included a lot of the files that were used by main but then they all include the global definitions instead of just using the ones at the top of main. That did give a speed up but is not optimum without removing all the extra includes.

Some time ago the main file had most of the machine in it and was a lot faster but was huge and hard to navigate. With other languages you don't get a huge penalty when you break the problem into smaller modules. OpenSCAD has a massive bug where calling a function or a module in a used file recalculates all the constants. I think it only needs to calculate any that involve $variables, (directly or indirectly) that have changed since the last time they were evaluated and that would affect the function or module being called. So not an easy bug to fix.
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

jon_bondy

Again, I wonder if this "hack" could be automated so that the OpenSCAD compiler generated this or reformulated a function without explicit user intervention.  Or some sort of pre-processor.

Sad that one must even consider this

Jon

On 6/15/2020 10:24 AM, nop head wrote:
I knocked 5 seconds of with this hack:

function shelf_screw_positions() = !is_undef($shelf_screw_positions) ? $shelf_screw_positions :
let(sw = spine_width(), sd = spine_depth(), bc = box_intrusion(box_type) + box_bezel_clearance(box_type)) [
    translate([z_axis_x + sw / 2, box_depth / 2 - sd + spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0, 90]),
    translate([z_axis_x + sw / 2, box_depth / 2 - shelf_boss_r - eps])                        * rotate([-90, 0, 90]),

    translate([-box_width / 4,               box_depth / 2])                                  * rotate([-90, 0,  0]),

    translate([box_width / 2,                box_depth / 2 - bc - shelf_boss_r - eps])        * rotate([-90, 0, -90]),
    translate([box_width / 2,               -box_depth / 2 + bc + shelf_boss_r + eps])        * rotate([-90, 0, -90]),

    translate([ 10, -box_depth / 2])                                                          * rotate([-90, 0, 180]),
    translate([-10, -box_depth / 2])                                                          * rotate([-90, 0, 180]),

    translate([-box_width / 2,               box_depth / 2 - bc - shelf_boss_r - eps])        * rotate([-90, 0, 90]),
    translate([-box_width / 2,              -box_depth / 2 + bc + shelf_boss_r + eps])        * rotate([-90, 0, 90]),

    translate([z_axis_x - sw / 2, box_depth / 2 - sd + spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0, -90]),
    translate([z_axis_x - sw / 2, box_depth / 2 - shelf_boss_r - eps])                        * rotate([-90, 0, -90]),
];


In main.scad I have:
$shelf_screw_positions = shelf_screw_positions();

The end effect is it only gets computed once in main and then any other call gets the value cached in the $variable.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

tp3
On 15.06.20 21:17, jon wrote:
> Again, I wonder if this "hack" could be automated so that
> the OpenSCAD compiler generated this or reformulated a
> function without explicit user intervention.  Or some sort
> of pre-processor.

To me it seems more useful to fix the root cause.

ciao,
  Torsten.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

nophead
Each constant assignment would need to track if its value depended on special variables or a function that depends on special variables. I suspect most don't, in which a big optimisation would be to skip re-evaluating them.

On Mon, 15 Jun 2020 at 20:23, Torsten Paul <[hidden email]> wrote:
On 15.06.20 21:17, jon wrote:
> Again, I wonder if this "hack" could be automated so that
> the OpenSCAD compiler generated this or reformulated a
> function without explicit user intervention.  Or some sort
> of pre-processor.

To me it seems more useful to fix the root cause.

ciao,
  Torsten.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

jon_bondy

I only made my [ridiculous] suggestion because I believe that this performance problem has been present for a long time, implying that fixing the root cause is not likely in the near term

Jon

On 6/15/2020 3:46 PM, nop head wrote:
Each constant assignment would need to track if its value depended on special variables or a function that depends on special variables. I suspect most don't, in which a big optimisation would be to skip re-evaluating them.

On Mon, 15 Jun 2020 at 20:23, Torsten Paul <[hidden email]> wrote:
On 15.06.20 21:17, jon wrote:
> Again, I wonder if this "hack" could be automated so that
> the OpenSCAD compiler generated this or reformulated a
> function without explicit user intervention.  Or some sort
> of pre-processor.

To me it seems more useful to fix the root cause.

ciao,
  Torsten.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Problem with large projects

tp3
On 15.06.20 21:50, jon wrote:
> I only made my [ridiculous] suggestion because I believe
> that this performance problem has been present for a long
> time, implying that fixing the root cause is not likely
> in the near term

I think the chance for a specific issue to be fixed has
not much correlation to how old the issue is.

In my opinion the chances depend on how complicated it seems
to be and/or how difficult it is to reproduce (= anticipated
time needed to get it fixed) and the amount of time and
motivation people have available.

Example with some really nice work:
https://github.com/openscad/openscad/pull/3351

ciao,
  Torsten.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten