Problem with large projects

Yes this has really become a problem for me. I have a list of transformations that position screws in a shelf bracket that is reasonably complex to calculate and since it generates the bracket and also positions inserts and screws into it plus it needs to drill holes in 6 panels that it mates with I need to reference it many times.

function shelf_screw_positions() = [
    translate([z_axis_x + spine_width() / 2, box_depth / 2 - spine_depth() + spine_clearance + shelf_boss_r + eps])               * rotate([-90, 0, 90]),
    translate([z_axis_x + spine_width() / 2, box_depth / 2 - shelf_boss_r - eps])                                                 * rotate([-90, 0, 90]),

    translate([-box_width / 4,               box_depth / 2])                                                                      * rotate([-90, 0,  0]),

    translate([box_width / 2,                box_depth / 2 - box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) - eps]) * rotate([-90, 0, -90]),
    translate([box_width / 2,               -box_depth / 2 + box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) + eps]) * rotate([-90, 0, -90]),

    translate([ 10, -box_depth / 2]) * rotate([-90, 0, 180]),
    translate([-10, -box_depth / 2]) * rotate([-90, 0, 180]),

    translate([-box_width / 2,               box_depth / 2 - box_intrusion(box_type) - shelf_boss_r - box_bezel_clearance(box_type) - eps]) * rotate([-90, 0, 90]),
    translate([-box_width / 2,              -box_depth / 2 + box_intrusion(box_type) + shelf_boss_r + box_bezel_clearance(box_type) + eps]) * rotate([-90, 0, 90]),

    translate([z_axis_x - spine_width() / 2, box_depth / 2 - spine_depth() + spine_clearance + shelf_boss_r + eps])               * rotate([-90, 0, -90]),
    translate([z_axis_x - spine_width() / 2, box_depth / 2 - shelf_boss_r - eps])                                                 * rotate([-90, 0, -90]),
];

If I store it in a variable it gets calculated every time I call a function or module in a file that includes its definition. If I wrap it in a function it gets computed every time I use it. It is bonkers because it is a constant, so only needs to be calculated once.

Adding this shelf to my project doubles the time it takes to do "Compiling design (CSG Tree generation)...", from about 30 seconds to 1 minute simply because this self touches six panels, which are all separate files in the machine. Compared to the rest of the complexity of the machine it is trivial but because it interacts with most of it it has a huge impact on the speed and the time needed to see every change.

Is it true that the only way to have a constant calculated once is to make my project into one massive file?

On Sun, 7 Jun 2020 at 21:03, nop head <[hidden email]> wrote:

Yes I remember that post. I seem to get away with it on all my projects except this one. It also has a global file with constants at the project level and they are positions of things with several function calls each to other modules, so they are more costly. Each of the function calls likely needs all the constants instantiated again and it explodes.

I can get the time down to 37 seconds by including sub assemblies instead of using them but I had to add a bodge to stop them instantiating themselves as I like to load a sub assembly when working on it.

I am on Windows so to profile OpenSCAD I think I would need to build it with Visual Studio, which I have done before, but I am pretty sure the problem is evaluating constants over and over again. Every single screw, nut, washer, etc are calls to the library and the library includes its global constants in every module as again, they all work standalone.  

It is counter intuitive that making constants in a file module is slower than working everything out inside functions. You would expect the constant to be evaluated once when the file is first used, not every time a function or module is called in it.

It might be faster to make all constants into functions but then you would have a big tree of function calls that explode as nothing can be worked out once and stored for later use.

On Sun, 7 Jun 2020 at 18:33, adrianv <[hidden email]> wrote:

Yeah, when I posted about this a few months ago people seemed to think it
didn't matter.  The issue has been open for years and seems catastrophic to
me. 

Our solution is to never use "use" and only use "include" which avoids the
problem.  I think there are other cases where "use" is slow as well, though
not as serious. 




--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

Yes that is the problem. If you have some complex code that generates a solid it is all run on every update, the parameters it generates are then compared with the cached solids to see if the geometry needs updating. In my project that takes no time at all but running all the code to generate thousands of parts starts to take a long time. I think this is mainly because every constant that isn't in your top level file gets evaluated over and over again. 

For isolation I have individual files that I can load on their own, for example the z axis assembly, which is quite complex. At the start of the file it has a lot of constants to work out positions, etc. It also includes definitions of all the vitamins it uses, which are hundreds of lists.  It also needs to export a few functions and modules to the rest of the machine, which just export constants, like the total height. However calling any of these means all the hundreds of constants it uses are all evaluated over and over again. So even on a PC it starts to take significant time. It really shouldn't take that long to evaluate a script but the constant evaluation explosion does as soon as you separate a project into lots of files.

On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano <[hidden email]> wrote:

in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?

As far as I know, they will not be cached. Only solids, models, objects are cached. Variables are not.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

Would there be any advantage to temporarily linearizing (combining) the files just for the purpose of compilation?  Many people run external editors and only use OpenSCAD as a compiler.  Suppose the editor automatically created a file called Linearized.SCAD and that was what OpenSCAD focused on, leaving the structure of your source code intact for editing.  Quite a hack, but would there be any benefits?

Yes that is the problem. If you have some complex code that generates a solid it is all run on every update, the parameters it generates are then compared with the cached solids to see if the geometry needs updating. In my project that takes no time at all but running all the code to generate thousands of parts starts to take a long time. I think this is mainly because every constant that isn't in your top level file gets evaluated over and over again. 

For isolation I have individual files that I can load on their own, for example the z axis assembly, which is quite complex. At the start of the file it has a lot of constants to work out positions, etc. It also includes definitions of all the vitamins it uses, which are hundreds of lists.  It also needs to export a few functions and modules to the rest of the machine, which just export constants, like the total height. However calling any of these means all the hundreds of constants it uses are all evaluated over and over again. So even on a PC it starts to take significant time. It really shouldn't take that long to evaluate a script but the constant evaluation explosion does as soon as you separate a project into lots of files.

On Mon, 15 Jun 2020 at 01:16, Ronaldo Persiano <[hidden email]> wrote:

in that library all transformations are just functiontions producing matrices, and a sequence of transformations is easily accumulated into a single multmatrix by the library, so essentialy all your code below will break down into a single multmatrix operation in the CSG tree. all the calculation itself is not passed CSG, it is just matrix multiplications, so essentially every combination of input variabled becomes one matrix value and i think this will be chached right?

As far as I know, they will not be cached. Only solids, models, objects are cached. Variables are not.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

Were you to manually put all of the files into a single temporary file, and compile that temporary file, what speed benefit would you see?  If the benefit is large, why not write a program to do all of the <include>s and <uses> automatically and then only use separate files for writing and the automatically generated temporary file  for compiling? 

Well I included a lot of the files that were used by main but then they all include the global definitions instead of just using the ones at the top of main. That did give a speed up but is not optimum without removing all the extra includes.

Some time ago the main file had most of the machine in it and was a lot faster but was huge and hard to navigate. With other languages you don't get a huge penalty when you break the problem into smaller modules. OpenSCAD has a massive bug where calling a function or a module in a used file recalculates all the constants. I think it only needs to calculate any that involve $variables, (directly or indirectly) that have changed since the last time they were evaluated and that would affect the function or module being called. So not an easy bug to fix.

http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

Again, I wonder if this "hack" could be automated so that the OpenSCAD compiler generated this or reformulated a function without explicit user intervention.  Or some sort of pre-processor.

Sad that one must even consider this

Jon

I knocked 5 seconds of with this hack:

function shelf_screw_positions() = !is_undef($shelf_screw_positions) ? $shelf_screw_positions :
let(sw = spine_width(), sd = spine_depth(), bc = box_intrusion(box_type) + box_bezel_clearance(box_type)) [
    translate([z_axis_x + sw / 2, box_depth / 2 - sd + spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0, 90]),
    translate([z_axis_x + sw / 2, box_depth / 2 - shelf_boss_r - eps])                        * rotate([-90, 0, 90]),

    translate([-box_width / 4,               box_depth / 2])                                  * rotate([-90, 0,  0]),

    translate([box_width / 2,                box_depth / 2 - bc - shelf_boss_r - eps])        * rotate([-90, 0, -90]),
    translate([box_width / 2,               -box_depth / 2 + bc + shelf_boss_r + eps])        * rotate([-90, 0, -90]),

    translate([ 10, -box_depth / 2])                                                          * rotate([-90, 0, 180]),
    translate([-10, -box_depth / 2])                                                          * rotate([-90, 0, 180]),

    translate([-box_width / 2,               box_depth / 2 - bc - shelf_boss_r - eps])        * rotate([-90, 0, 90]),
    translate([-box_width / 2,              -box_depth / 2 + bc + shelf_boss_r + eps])        * rotate([-90, 0, 90]),

    translate([z_axis_x - sw / 2, box_depth / 2 - sd + spine_clearance + shelf_boss_r + eps]) * rotate([-90, 0, -90]),
    translate([z_axis_x - sw / 2, box_depth / 2 - shelf_boss_r - eps])                        * rotate([-90, 0, -90]),
];

In main.scad I have:

$shelf_screw_positions = shelf_screw_positions();

The end effect is it only gets computed once in main and then any other call gets the value cached in the $variable.

I only made my [ridiculous] suggestion because I believe that this performance problem has been present for a long time, implying that fixing the root cause is not likely in the near term

Jon

Each constant assignment would need to track if its value depended on special variables or a function that depends on special variables. I suspect most don't, in which a big optimisation would be to skip re-evaluating them.

On Mon, 15 Jun 2020 at 20:23, Torsten Paul <[hidden email]> wrote:

On 15.06.20 21:17, jon wrote:
> Again, I wonder if this "hack" could be automated so that
> the OpenSCAD compiler generated this or reformulated a
> function without explicit user intervention.  Or some sort
> of pre-processor.

To me it seems more useful to fix the root cause.

ciao,
  Torsten.